Automatic display of previously-acquired endoluminal images

ABSTRACT

Apparatus and methods are provided for use with an endoluminal data-acquisition device that acquires a set of endoluminal data-points of a lumen of a subject&#39;s body at respective locations inside the lumen, a second endoluminal device, and a display configured to display images. At least one processor includes location-association functionality that associates a given data point acquired by the endoluminal data-acquisition device with a given location within the lumen. Location-determination functionality determines, by means of image processing, in an extraluminal image of the second endoluminal device, a current location of at least a portion of the second endoluminal device. Display-driving functionality drives the display to display an indication of the endoluminal data point associated with the location, in response to determining that the portion of the second device is currently at the location. Other applications are also described.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 13/228,185, filed Sep. 8, 2011, now U.S. Pat. No.10,716,528, which is a continuation of U.S. patent application Ser. No.12/666,879, filed Mar. 29, 2012, now U.S. Pat. No. 8,781,193 and acontinuation of International Application No. PCT/IL2011/000612,entitled “Co-use of endoluminal data and extraluminal imaging,” filed 28Jul. 2011, which:

(a) claims the benefit of:

U.S. Provisional Patent Application 61/344,464, entitled “Co-use ofendoluminal data and extraluminal imaging,” filed 29 Jul. 2010;

U.S. Provisional Patent Application 61/344,875, entitled “Co-use ofendoluminal data and extraluminal imaging,” filed 1 Nov. 2010;

U.S. Provisional Patent Application 61/457,339, entitled “Co-use ofendoluminal data and extraluminal imaging,” filed 3 Mar. 2011;

U.S. Provisional Patent Application 61/457,455, entitled “Co-use ofendoluminal data and extraluminal imaging,” filed 1 Apr. 2011;

U.S. Provisional Patent Application 61/457,780, entitled “Co-use ofendoluminal data and extraluminal imaging,” filed 2 Jun. 2011; and

U.S. Provisional Patent Application 61/457,951, entitled “Co-use ofendoluminal data and extraluminal imaging,” filed 15 Jul. 2011; and

(b) is a continuation-in-part of U.S. patent application Ser. No.12/650,605 to Cohen (published as US 2010/0172556), filed Dec. 31, 2009,which:

(i) is a continuation of U.S. patent application Ser. No. 12/666,879 toSteinberg, filed Mar. 29, 2012, which is the US national phase of PCTApplication No. PCT/IL2009/001089 to Cohen (published as WO 10/058398),filed Nov. 18, 2009, which claims priority from the following patentapplications:

-   -   U.S. Provisional Patent Application 61/193,329, entitled        “Apparatuses and methods for the automatic generation of a road        map from angiographic images of a cyclically-moving organ,” to        Steinberg, filed Nov. 18, 2008    -   U.S. Provisional Patent Application 61/193,915, entitled “Image        processing and tool actuation for medical procedures,” to        Steinberg, filed Jan. 8, 2009    -   U.S. Provisional Patent Application 61/202,181, entitled “Image        processing and tool actuation for medical procedures,” to        Steinberg, filed Feb. 4, 2009    -   U.S. Provisional Patent Application 61/202,451, entitled “Image        processing and tool actuation for medical procedures,” to        Steinberg, filed Mar. 2, 2009    -   U.S. Provisional Patent Application 61/213,216, entitled “Image        processing and tool actuation for medical procedures,” to        Steinberg, filed May 18, 2009    -   U.S. Provisional Patent Application 61/213,534, entitled “Image        Processing and Tool Actuation for Medical Procedures,” to        Steinberg, filed Jun. 17, 2009    -   U.S. Provisional Patent Application 61/272,210, entitled “Image        processing and tool actuation for medical procedures,” to        Steinberg, filed Sep. 1, 2009 and    -   U.S. Provisional Patent Application 61/272,356, entitled “Image        Processing and Tool Actuation for Medical Procedures” to        Steinberg, filed Sep. 16, 2009; and

(ii) is a continuation-in-part of U.S. patent application Ser. No.12/075,244 to Tolkowsky (published as US 2008/0221442), filed Mar. 10,2008, entitled “Imaging for use with moving organs,” which claims thebenefit of U.S. Provisional Patent Application Nos.:

-   -   60/906,091 filed on Mar. 8, 2007,    -   60/924,609 filed on May 22, 2007,    -   60/929,165 filed on Jun. 15, 2007,    -   60/935,914 filed on Sep. 6, 2007, and    -   60/996,746 filed on Dec. 4, 2007,    -   all entitled “Apparatuses and methods for performing medical        procedures on cyclically-moving body organs.”

The present application is related to the following patent applications:

-   -   U.S. patent application Ser. No. 12/075,214 to Iddan (published        as 2008/0221439), filed Mar. 10, 2008, entitled “Tools for use        with moving organs.”    -   U.S. patent application Ser. No. 12/075,252 to Iddan (published        as US 2008/0221440), filed Mar. 10, 2008, entitled “Imaging and        tools for use with moving organs.”    -   U.S. patent application Ser. No. 12/781,260 to Blank (published        as US 2010/0228076), filed May 17, 2010, entitled “Controlled        actuation and deployment of a medical device.”    -   U.S. patent application Ser. No. 12/487,315 to Iddan (published        as US 2009/0306547), filed Jun. 18, 2009, entitled “Stepwise        advancement of a medical tool,” which claims the benefit of U.S.        Provisional Patent Application No. 61/129,331 to Iddan, filed on        Jun. 19, 2008, entitled “Stepwise advancement of a medical        tool.”

All of the above-mentioned applications are incorporated herein byreference.

FIELD OF EMBODIMENTS OF THE INVENTION

Some applications of the present invention generally relate to medicalimaging. Specifically, some applications of the present invention relateto the co-use of endoluminal data and extraluminal imaging.

BACKGROUND

Vascular catheterizations, such as coronary catheterizations, arefrequently-performed medical interventions. Such interventions aretypically performed in order to diagnose the blood vessels for potentialdisease, and/or to treat diseased blood vessels. Typically, in order tofacilitate diagnosis of blood vessels, the catheterization is performedunder extraluminal imaging. For some procedures, an endoluminaldata-acquisition device is used to perform endoluminal imaging and/ormeasurements. If appropriate based on the diagnosis, a treatment isapplied to the blood vessel. For some procedures, treatment of the bloodvessel includes the application of a treatment to the blood vessel by atherapeutic device that is placed endoluminally. For example, atherapeutic device (e.g., a balloon) is placed in the blood vesseltemporarily and retrieved subsequent to the treatment having beenapplied. Alternatively, a therapeutic device (e.g., a stent) may remainimplanted inside the blood vessel in order to treat the blood vessel.

SUMMARY OF EMBODIMENTS

Some applications of the present invention are applied to medicalprocedures performed, in whole or in part, on or within luminalstructures. For some applications, apparatus and methods are providedfor facilitating the co-use of extraluminal imaging and endoluminal data(i.e., data that are acquired using an endoluminal data-acquisitiondevice), in performing medical procedures. Endoluminal data may includeimaging data (e.g., imaging data acquired using an endoluminal imagingprobe), data derived from measurements (e.g., measurements performedusing an endoluminal sensor or measuring device), other data, and anycombination thereof.

In accordance with some applications of the present invention, duringinsertion and deployment of an endoluminal device, e.g., an endoluminaltherapeutic device, into a lumen, real-time extraluminal images of thedevice inside the lumen are displayed together with endoluminal datathat were acquired previously and that correspond to the currentlocation of the endoluminal therapeutic device. The cumulative effect ofshowing the extraluminal images and the endoluminal data is as if theendoluminal therapeutic tool is being inserted and deployed under bothextraluminal imaging and endoluminal data acquisition. For someapplications, the aforementioned techniques are applied since it isdifficult or impossible to acquire the endoluminal data during insertionand deployment of the therapeutic device, because the lumen is toonarrow to accommodate both the endoluminal therapeutic device and theendoluminal data-acquisition device. Alternatively, although it may bepossible for the lumen to accommodate both the endoluminal therapeuticdevice and the endoluminal data-acquisition device, the aforementionedtechniques may be used to prevent the endoluminal data-acquisitiondevice from interfering with the endoluminal therapeutic device, duringinsertion and/or deployment of the therapeutic device.

There is therefore provided, in accordance with some applications of thepresent invention, apparatus for use with an endoluminaldata-acquisition device that is configured to acquire a set ofendoluminal data-points with respect to a lumen of a body of a subjectat respective locations inside the lumen, a second endoluminal device,and a display configured to display images of the lumen, the apparatusincluding:

at least one processor, including:

-   -   location-association functionality configured to associate a        given endoluminal data point acquired by the endoluminal        data-acquisition device with a given location within the lumen;    -   location-determination functionality configured, in an        extraluminal image of the second endoluminal device, to        determine by means of image processing, a current location of at        least a portion of the second endoluminal device inside the        lumen;    -   display-driving functionality configured, in response to        determining that the portion of the second endoluminal device is        currently at the given location, to drive the display to display        an indication of the endoluminal data point associated with the        given location.

For some applications, the second endoluminal device includes a secondendoluminal data-acquisition device configured to acquire a second setof endoluminal data-points with respect to the lumen at respectivelocations inside the lumen, and the display-driving functionality isconfigured, in response to determining that the portion of the secondendoluminal data-acquisition device is currently at the given location,to drive the display to display:

an endoluminal image acquired by the first endoluminal data-acquisitiondevice that corresponds to the given location, and

an endoluminal image acquired by the second endoluminal data-acquisitiondevice that corresponds to the given location.

For some applications, the second endoluminal device includes a secondendoluminal data-acquisition device configured to acquire a second setof endoluminal data-points with respect to the lumen at respectivelocations inside the lumen, and the display-driving functionality isconfigured, in response to determining that the portion of the secondendoluminal data-acquisition device is currently at the given location,to drive the display to display:

an endoluminal image acquired by the second endoluminal data-acquisitiondevice that corresponds to the given location, and

an indication of the given location with respect to an endoluminal imagestack of the lumen generated using the endoluminal data points acquiredby the first endoluminal data-acquisition device.

For some applications, the endoluminal data-acquisition device includesan endoluminal imaging probe configured to acquire endoluminal images ofthe lumen at respective locations inside the lumen, and thelocation-association functionality is configured to associate a givenendoluminal image acquired by the endoluminal imaging probe with a givenlocation within the lumen.

For some applications, the display-driving functionality is configured,in response to determining that the portion of the second endoluminaldevice is currently at the given location, to drive the display todisplay an endoluminal image that corresponds to the given location.

For some applications, the display-driving functionality is configured,in response to determining that the portion of the second endoluminaldevice is currently at the given location, to drive the display todisplay an indication of the given location with respect to anendoluminal image stack of the lumen.

For some applications, the display-driving functionality is configured,in response to determining that the portion of the second endoluminaldevice is currently at the given location, to drive the display todisplay an indication of the given location with respect to theextraluminal image of the lumen.

For some applications, the second endoluminal device includes a secondendoluminal data-acquisition device configured to acquire a second setof endoluminal data-points with respect to the lumen at respectivelocations inside the lumen, and the display-driving functionality isfurther configured, in response to determining that the portion of thesecond endoluminal data-acquisition device is currently at the givenlocation, to drive the display to display:

an endoluminal image acquired by the first endoluminal data-acquisitiondevice that corresponds to the given location, and

an endoluminal image acquired by the second endoluminal data-acquisitiondevice that corresponds to the given location.

For some applications, the second endoluminal device includes a secondendoluminal data-acquisition device configured to acquire a second setof endoluminal data-points with respect to the lumen at respectivelocations inside the lumen, and the display-driving functionality isfurther configured, in response to determining that the portion of thesecond endoluminal data-acquisition device is currently at the givenlocation, to drive the display to display:

an endoluminal image acquired by the second endoluminal data-acquisitiondevice that corresponds to the given location, and

an indication of the given location with respect to an endoluminal imagestack of the lumen generated using the endoluminal data points acquiredby the first endoluminal data-acquisition device.

For some applications,

the endoluminal data-acquisition device includes a portion that isvisible in extraluminal images of the data-acquisition device inside thelumen, and

the location-association functionality is configured to associate theendoluminal data point with the given location inside the lumen bydetermining, by means of image-processing, in an extraluminal image ofthe data-acquisition device inside the lumen, a location of at least thevisible portion of the data-acquisition device inside the lumen, at theacquisition of the endoluminal data point.

For some applications,

the endoluminal data-acquisition device includes an image-acquiringportion, and

the location-association functionality is configured to associate theendoluminal data point with the given location inside the lumen byaccounting for an offset between the portion of the endoluminaldata-acquisition device that is visible in the extraluminal image, andthe image-acquiring portion of the endoluminal data-acquisition device.

For some applications, the second endoluminal device includes anendoluminal therapeutic device configured to apply a therapy to thelumen, and the location-determination functionality is configured, in anextraluminal image of the endoluminal therapeutic device, to determineby means of image processing, a current location of at least a portionof the endoluminal therapeutic device inside the lumen.

For some applications, the endoluminal therapeutic device includes aguidewire configured to penetrate an occlusion of the lumen and theendoluminal data-acquisition device includes a forward-lookingendoluminal imaging probe, and the location-association functionalityconfigured to associate the given endoluminal data point with the givenlocation by associating an endoluminal image of a portion of the lumenthat is distal to the given location with the given location.

There is further provided, in accordance with some applications of thepresent invention, a method, including:

acquiring a set of endoluminal data point of a lumen of a subject'sbody;

determining that one of the endoluminal data points of the setcorresponds to a given location inside the lumen; and

subsequently,

while a second endoluminal device is inside the lumen:

-   -   acquiring an extraluminal image of the second endoluminal device        inside the lumen;    -   by means of image processing, determining, based upon the        extraluminal image, a current location of at least a portion of        the second endoluminal device inside the lumen; and    -   in response to determining that the portion of the second        endoluminal device is currently at the given location,        displaying an indication of the endoluminal data point that        corresponds to the given location.

For some applications, the second endoluminal device includes anendoluminal therapeutic device configured to apply a therapy to thelumen, and acquiring the extraluminal image of the second endoluminaldevice inside the lumen includes acquiring an extraluminal image of theendoluminal therapeutic device inside the lumen.

For some applications, the endoluminal therapeutic device includes aguidewire, and the method further includes penetrating an occlusion ofthe lumen with the guidewire.

For some applications, acquiring the at least one endoluminal data pointincludes, while a forward-looking endoluminal imaging probe is at thegiven location, acquiring an endoluminal image of a portion of the lumenthat is distal to the given location.

There is additionally provided, in accordance with some applications ofthe present invention, a method for use with an endoluminaldata-acquisition device configured to be moved through a lumen of asubject's body, the endoluminal data-acquisition device having aradiopaque marker coupled thereto, including:

while the endoluminal data-acquisition device is being moved through thelumen, acquiring a plurality of endoluminal data points of the lumenusing the endoluminal data-acquisition device;

determining that a first endoluminal data point corresponds to a firstlocation within the lumen, by:

-   -   acquiring a first angiographic image of the lumen, at a time        associated with an acquisition of the first endoluminal data        point by the endoluminal data-acquisition device, and    -   determining a location of the radiopaque marker within the first        angiographic image of the lumen, by performing image processing        on the first angiographic image, the location of the radiopaque        marker within the first angiographic image of the lumen        corresponding to the first endoluminal data point;

determining that a second endoluminal data point corresponds to a secondgiven location within the lumen, by:

-   -   acquiring a second angiographic image of the lumen, at a time        associated with an acquisition of the second endoluminal data        point by the endoluminal data-acquisition device, and    -   determining a location of the radiopaque marker within the        second angiographic image of the lumen by performing image        processing on the second angiographic image, the location of the        radiopaque marker within the second angiographic image of the        lumen corresponding to the second endoluminal data point;

generating a combined angiographic image of the lumen that includesrepresentations of the first and second marker locations thereon, byco-registering the first and second angiographic images; and

determining that at least one location on the combined angiographicimage that is intermediate to the first and second locations of theradiopaque marker corresponds to an endoluminal data point acquiredbetween the acquisitions of the first and second data points, byinterpolating between the first and second locations of the radiopaquemarker on the combined angiographic image; and

generating an output in response thereto.

For some applications, acquiring the plurality of endoluminal datapoints of the lumen using the endoluminal data-acquisition device whilethe endoluminal data-acquisition device is being moved through the lumenincludes acquiring the plurality of endoluminal data points of the lumenusing the endoluminal data-acquisition device while the endoluminaldata-acquisition device is being pulled-back through the lumen.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use with:

an endoluminal data-acquisition device configured to acquire a pluralityof endoluminal data points of a lumen of a body of a subject atrespective locations inside the lumen, while the endoluminaldata-acquisition device is moved through the lumen, the endoluminaldata-acquisition device having a radiopaque marker coupled thereto,

an angiographic imaging device configured to (a) acquire a firstangiographic image of the lumen, at a time associated with anacquisition of a first endoluminal data point by the endoluminaldata-acquisition device, and (b) acquire a second angiographic image ofthe lumen, at a time associated with an acquisition of a secondendoluminal data point by the endoluminal data-acquisition device, and

a display,

the apparatus including:

at least one processor, including:

-   -   location-determination functionality configured to:        -   determine that the first endoluminal data point corresponds            to a first location within the lumen, by determining a            location of the radiopaque marker within the first            angiographic image of the lumen, by performing image            processing on the first angiographic image, the location of            the radiopaque marker within the first angiographic image of            the lumen corresponding to the first endoluminal data point,            and        -   determine that a second endoluminal data point corresponds            to a second given location within the lumen by determining a            location of the radiopaque marker within the second            angiographic image of the lumen by performing image            processing on the second angiographic image, the location of            the radiopaque marker within the second angiographic image            of the lumen corresponding to the second endoluminal data            point;    -   image-co-registration functionality configured to generate a        combined angiographic image of the lumen that includes        representations of the first and second marker locations        thereon, by co-registering the first and second angiographic        images;    -   location-association functionality configured to determine that        at least one location on the combined angiographic image that is        intermediate to the first and second locations of the radiopaque        marker corresponds to an endoluminal data point acquired between        the acquisitions of the first and second data points, by        interpolating between the first and second locations of the        radiopaque marker on the combined angiographic image;    -   display-driving functionality configured to drive the display to        display an output, in response to determining that the        intermediate location corresponds to the endoluminal data point        acquired between the acquisitions of the first and second data        points.

For some applications, the location-determination functionality isconfigured to:

determine that the first endoluminal data point corresponds to the firstlocation within the lumen by determining that the first endoluminal datapoint corresponds to a location in a vicinity of a first end of aluminal segment of interest, and

determine that the second endoluminal data point corresponds to thesecond location within the lumen by determining that the secondendoluminal data point corresponds to a location in a vicinity of asecond end of the luminal segment of interest.

For some applications,

the location-determination functionality is configured to:

-   -   determine that the first endoluminal data point corresponds to        the first location within the lumen by determining that the        first endoluminal data point corresponds to a location in a        vicinity of a first end of a luminal segment of interest, and    -   determine that the second endoluminal data point corresponds to        the second location within the lumen by determining that the        second endoluminal data point corresponds to a location between        the first end and a second end of the luminal segment of        interest, and

the angiographic imaging device includes an angiographic imaging devicethat is further configured to acquire a third angiographic image of thelumen, at a time associated with an acquisition of a third endoluminaldata point by the endoluminal data-acquisition device,

the location-determination functionality is further configured todetermine that third endoluminal data point corresponds to a location ina vicinity of the second end of the luminal segment of interest, bydetermining a location of the radiopaque marker within the thirdangiographic image of the lumen by performing image processing on thethird angiographic image, the location of the radiopaque marker withinthe third angiographic image of the lumen corresponding to the thirdendoluminal data point;

the image-co-registration functionality is further configured togenerate a representation of the third marker location on the combinedangiographic image, by co-registering the first, second, and thirdangiographic images; and

the location-association functionality is further configured todetermine that at least one location on the combined angiographic imagethat is intermediate to the second and third locations of the radiopaquemarker corresponds to an endoluminal data point acquired between theacquisitions of the second and third data points, by interpolatingbetween the second and third locations of the radiopaque marker on thecombined angiographic image; and

the display-driving functionality is further configured to drive thedisplay to display an output, in response to determining that theintermediate location corresponds to the endoluminal data point acquiredbetween the acquisitions of the second and third data points.

For some applications, the location-association functionality isconfigured to interpolate between the first and second locations of theradiopaque marker on the combined angiographic image by assuming that,between acquiring respective successive pairs of endoluminal data pointsbetween the acquisitions of the first and second data points, theendoluminal data acquisition device traveled equal distances.

For some applications, the location-association functionality isconfigured to interpolate between the first and second locations of theradiopaque marker on the combined angiographic image by assuming that arate of the movement of the endoluminal data acquisition device waslinear between the acquisitions of the first and second data points.

There is additionally provided, in accordance with some applications ofthe present invention, a method for use with an endoluminaldata-acquisition device configured to be moved through a lumen of asubject's body, the endoluminal data-acquisition device having aradiopaque marker coupled thereto, including:

while the endoluminal data-acquisition device is being moved through thelumen:

-   -   acquiring a plurality of endoluminal data points of the lumen        using the endoluminal data-acquisition device;    -   continuously injecting contrast agent into the lumen; and    -   acquiring a plurality of angiographic images of the        data-acquisition device;

determining that endoluminal data points correspond to respectivelocations within the lumen, by determining locations of the radiopaquemarker within the angiographic images of the lumen, by performing imageprocessing on the angiographic images, the locations of the radiopaquemarker within the angiographic images of the lumen corresponding torespective endoluminal data points; and

generating an output in response thereto.

For some applications, continuously injecting the contrast agent intothe lumen includes continuously injecting the contrast agent into thelumen for a period of at least two seconds.

For some applications, acquiring the plurality of endoluminal datapoints of the lumen using the endoluminal data-acquisition deviceincludes acquiring the plurality of endoluminal data points of the lumenusing the endoluminal data-acquisition device while the data-acquisitiondevice is being pulled back through the lumen.

For some applications, continuously injecting the contrast agent intothe lumen includes continuously injecting the contrast agent over atleast 50% of a duration of a period over which the endoluminaldata-acquisition device acquires the endoluminal data points.

For some applications, continuously injecting the contrast agent intothe lumen includes continuously injecting the contrast agent over atleast 80% of a duration of a period over which the endoluminaldata-acquisition device acquires the endoluminal data points.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use with:

an endoluminal data-acquisition device configured to acquire a pluralityof endoluminal data points of a lumen of a body of a subject atrespective locations inside the lumen, while the endoluminaldata-acquisition device is being moved through the lumen, theendoluminal data-acquisition device having a radiopaque marker coupledthereto,

contrast agent configured to be continuously injected into the lumen,during the movement of the endoluminal data-acquisition device,

an angiographic imaging device configured to acquire a plurality ofangiographic images of the endoluminal data-acquisition device insidethe lumen, during the movement of the endoluminal data-acquisitiondevice, and

a display configured to display images of the lumen,

the apparatus including:

at least one processor, including:

-   -   location-association functionality configured to determine that        endoluminal data points correspond to respective locations        within the lumen, by determining locations of the radiopaque        marker within the angiographic images of the lumen, by        performing image processing on the angiographic images, the        locations of the radiopaque marker within the angiographic        images of the lumen corresponding to respective endoluminal data        points;    -   display-driving functionality configured to drive the display to        display an output, in response to determining that the        endoluminal data points correspond to respective locations        within the lumen.

For some applications, the endoluminal data-acquisition device includesan endoluminal imaging probe configured to acquire a plurality ofendoluminal images at a first frame rate, the angiographic imagingdevice includes an angiographic imaging device that is configured toacquire the plurality of angiographic images at a second frame rate thatis different from the first frame rate, and the location-associationfunctionality is configured to determine that endoluminal data pointscorrespond to respective locations within the lumen by indexing theendoluminal images with respect to the angiographic images.

There is additionally provided, in accordance with some applications ofthe present invention, a method for use with an endoluminaldata-acquisition device configured to be moved through a lumen of asubject's body, the endoluminal data-acquisition device having aradiopaque marker coupled thereto, including:

while the endoluminal data-acquisition device is being moved through thelumen, acquiring a plurality of endoluminal data points of the lumenusing the endoluminal data-acquisition device;

determining that respective endoluminal data points correspond torespective locations within the lumen, by acquiring at least first andsecond angiographic images of the lumen, and determining first andsecond locations of the marker respectively within the first and secondangiographic images;

generating a combined angiographic image of the lumen that includesrepresentations thereon of the first and second marker locations withinthe lumen, by co-registering the first and second angiographic images toone another, by:

-   -   designating one of the angiographic images as a baseline image,        a shape of the lumen in the baseline image being designated as a        baseline shape of the lumen;    -   determining whether a shape of the lumen in the angiographic        image that is not the baseline image is the same as the baseline        shape of the lumen; and    -   in response to determining that the shape of the lumen in the        angiographic image that is not the baseline image is not the        same as the baseline shape of the lumen:        -   designating the image that is not the baseline image as a            non-baseline image, and        -   deforming the shape of the lumen in the non-baseline image,            such that the shape of the lumen becomes more similar to the            baseline shape of the portion than when the lumen in the            non-baseline image is not deformed;        -   based upon the deformation of the non-baseline image,            determining a location upon the baseline image at which the            marker from within the non-baseline image should be located;            and        -   generating an indication of the marker from within the            non-baseline image at the determined location on the            baseline image.

For some applications, acquiring the plurality of endoluminal datapoints of the lumen using the endoluminal data-acquisition device whilethe endoluminal data-acquisition device is being moved through the lumenincludes acquiring the plurality of endoluminal data points of the lumenusing the endoluminal data-acquisition device while the endoluminaldata-acquisition device is being pulled-back through the lumen.

There is additionally provided, in accordance with some applications ofthe present invention, apparatus for use with:

an endoluminal data-acquisition device configured to acquire a pluralityof endoluminal data points of a lumen of a body of a subject atrespective locations inside the lumen, while the endoluminaldata-acquisition device is moved through the lumen, the endoluminaldata-acquisition device having a radiopaque marker coupled thereto,

an angiographic imaging device configured to acquire respectiveangiographic image of the lumen, at times associated with acquisitionsof respective endoluminal data point by the endoluminal data-acquisitiondevice, and

a display,

the apparatus including:

at least one processor, including:

-   -   location-determination functionality configured to determine        first and second locations of the radiopaque marker respectively        within first and second angiographic images of the lumen;    -   image-co-registration functionality configured to generate a        combined angiographic image of the lumen that includes        representations of the first and second marker locations        thereon, by co-registering the first and second angiographic        images to one another, by:        -   designating one of the angiographic images as a baseline            image, a shape of the lumen in the baseline image being            designated as a baseline shape of the lumen;        -   determining whether a shape of the lumen in the angiographic            image that is not the baseline image is the same as the            baseline shape of the lumen; and        -   in response to determining that the shape of the lumen in            the angiographic image that is not the baseline image is not            the same as the baseline shape of the lumen:            -   designating the image that is not the baseline image as                a non-baseline image, and            -   deforming the shape of the lumen in the non-baseline                image, such that the shape of the lumen becomes more                similar to the baseline shape of the portion than when                the lumen in the non-baseline image is not deformed;            -   based upon the deformation of the non-baseline image,                determining a location upon the baseline image at which                the marker from within the non-baseline image should be                located; and            -   generating an indication of the marker from within the                non-baseline image at the determined location on the                baseline image; and        -   display-driving functionality configured to drive the            display to display an output, in response to generating the            combined angiographic image of the lumen.

For some applications, the location-determination functionality isconfigured to:

determine that a first endoluminal data point corresponds to a firstlocation within the lumen, by determining a location of the radiopaquemarker within the first angiographic image of the lumen, by performingimage processing on the angiographic image, the location of the firstradiopaque marker within the first angiographic image of the lumencorresponding to the first endoluminal data point, and

determine that a second endoluminal data point corresponds to a secondgiven location within the lumen by determining a location of theradiopaque marker within the second angiographic image of the lumen byperforming image processing on the second angiographic image, thelocation of the radiopaque marker within the second angiographic imageof the lumen corresponding to the second endoluminal data point.

For some applications, the location-determination functionality isconfigured to:

determine that the first endoluminal data point corresponds to the firstlocation within the lumen by determining that the first endoluminal datapoint corresponds to a location in a vicinity of a first end of aluminal segment of interest, and

determine that the second endoluminal data point corresponds to thesecond location within the lumen by determining that the secondendoluminal data point corresponds to a location in a vicinity of asecond end of the luminal segment of interest.

For some applications, the at least one processor further includeslocation-association functionality configured to determine that at leastone location on the combined angiographic image that is intermediate tothe first and second locations of the radiopaque marker corresponds toan endoluminal data point acquired between the acquisitions of the firstand second data points, by interpolating between the first and secondlocations of the radiopaque marker on the combined angiographic image.

There is further provided, in accordance with some applications of thepresent invention, a method for imaging a tool inside a portion of abody of a subject that undergoes motion, the tool having contours, themethod including:

acquiring a plurality of image frames of the portion of the subject'sbody; and

generating at least one image frame in which the tool is enhanced, by:

-   -   identifying radiopaque markers in the image frames;    -   identifying edge lines in a vicinity of the markers within the        image frames, the edge lines corresponding to contours of the        tool;    -   in response to the identifying of the edge lines, selecting a        subset of the image frames that are based upon the acquired        image frames, based upon a level of similarity between the edge        lines in the selected image frames to one another;    -   aligning the contours in a plurality of the selected image        frames, and    -   averaging the plurality of aligned frames to generate an        averaged image frame; and

displaying the averaged image frame.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use with a tool configured to be placedinside a portion of a body of a subject that undergoes motion, the toolhaving contours, an image-acquisition device configured to acquire aplurality of image frames of the portion of the subject's body, and adisplay, the apparatus including:

at least one processor configured to generate at least one image framein which the tool is enhanced, the processor including:

-   -   image-receiving functionality configured to receive the        plurality of image frames into the processor,    -   marker-identifying functionality configured to automatically        identify radiopaque markers in the image frames,    -   edge-line-identifying functionality configured to automatically        identify edge lines in a vicinity of the radiopaque markers in        the image frames,    -   image-selection functionality configured, in response to the        identifying of the edge lines, to select a subset of the image        frames that are based upon the acquired image frames, based upon        a level of similarity between the edge lines in the selected        image frames to one another,    -   image-alignment functionality configured to align the edge lines        in a plurality of the selected image frames, and    -   image-averaging functionality configured to generate an averaged        image frame by averaging the plurality of aligned image frames;        and    -   display-driving functionality configured to drive the display to        display the averaged image frame.

For some applications, the image-selection functionality is configuredto select the subset of image frames based upon a level of similaritybetween shapes of the edge lines in the image frames.

For some applications, the image-selection functionality is configuredto select the subset of image frames based upon a level of alignmentbetween the edge lines and the radiopaque markers in the image frames.

For some applications, the image-selection functionality is configuredto select the subset of image frames by rejecting from being included inthe subset, at least one image frame in which edge lines correspondingto the contours of the tool appear.

For some applications, the image-alignment functionality is configuredto align the edge lines in the selected image frames by translating atleast one image frame with respect to at least one other image frame ofthe selected image frames.

For some applications, the processor is configured to generate aplurality of image frames in which the tool is enhanced, and thedisplay-driving functionality is configured to drive the display todisplay, as an image stream, the plurality of image frames in which thetool is enhanced.

For some applications, the tool includes a stent that is inserted intothe lumen while disposed on a device, and the marker-identifyingfunctionality is configured to identify the radiopaque markers byidentifying radiopaque markers that are coupled to the device, and theedge-line-identifying functionality is configured to identify the edgelines by identifying curved edge lines, corresponding to contours of thestent.

For some applications, the image-selection functionality is configuredto select the subset of image frames based upon a level of similaritybetween shapes of the curved edge lines in the image frames.

For some applications, the marker-identifying functionality isconfigured to identify first and second radiopaque markers that arecoupled to the device, and the image-selection functionality isconfigured to select the subset of image frames based upon a level ofalignment between the edge lines and an imaginary line running from thefirst marker to the second marker in the image frames.

There is further provided, in accordance with some applications of thepresent invention, a method for use with an endoluminal data-acquisitiondevice configured to acquire endoluminal data points while movingthrough a lumen of a subject's body generally in a first direction withrespect to the lumen, including:

while the endoluminal data-acquisition device is being moved through thelumen, acquiring a plurality of endoluminal data points of the lumenusing the endoluminal data-acquisition device;

determining that, at at least one location, two or more endoluminal datapoints were acquired;

generating an output using a portion of the plurality of endoluminaldata points of the lumen acquired using the endoluminal data-acquisitiondevice, by using only a single endoluminal data point corresponding tothe location.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use with an endoluminaldata-acquisition device that acquires a plurality of endoluminal datapoints of a lumen of a body of a subject while being moved through thelumen generally in a first direction with respect to the lumen, and adisplay, the apparatus including:

at least one processor including:

-   -   duplicate-data-point-identification functionality configured to        determine that, at at least one location, two or more        endoluminal data points were acquired by the endoluminal        data-acquisition device;    -   data-point-selection functionality configured to generate an        output using a portion of the plurality of endoluminal data        points of the lumen acquired using the endoluminal        data-acquisition device, by using only a single data point        corresponding to the location; and    -   display-driving functionality configured to drive the display to        display the output.

For some applications, the data-point-selection functionality isconfigured to use only the single data point corresponding to thelocation by using a single one of the two or more endoluminal datapoints that were acquired at the at least one location, and rejectinganother one of the two or more endoluminal data points from being usedin the output

For some applications, the data-point-selection functionality isconfigured to generate the output selecting for use as the single datapoint a data point that was acquired at the given location at anearliest time with respect to the data points that were acquired at thegiven location.

For some applications, the data-point-selection functionality isconfigured to generate the output by rejecting from being used in theoutput an endoluminal data point that was acquired while the device wasmoving in a second direction with respect to the lumen that is oppositeto the first direction.

For some applications, the data-point-selection functionality isconfigured:

to generate the output by generating an indication that one of the twoor more endoluminal data points is associated with the location byco-registering the portion of the plurality of endoluminal data pointsof the lumen with the extraluminal image, and

to reject the other one of the two or more endoluminal data points frombeing used in the output by rejecting one of the two or more endoluminalimages from being indicated to be associated with the location.

For some applications,

the endoluminal data-acquisition device includes an endoluminal imagingprobe configured to acquire a plurality of endoluminal image frames ofthe lumen, and

the data-point-selection functionality is configured:

-   -   to generate the output by generating an endoluminal image stack        using some of the plurality of endoluminal image frames of the        lumen, and    -   to reject the other one of the two or more endoluminal data        points from being used in the output by rejecting the other one        of the two or more endoluminal image frames from being displayed        in the image stack.

For some applications, the duplicate-data-point-identificationfunctionality is configured to determine that, at at least one location,two or more endoluminal data points were acquired by determining thatthe endoluminal data-acquisition device moved past the location in asecond direction with respect to the lumen that is opposite to the firstdirection, and the data-point-selection functionality is configured togenerate the output by placing image frames in order within the imagestack based on determining that the endoluminal data-acquisition devicemoved past the location, in the second direction with respect to thelumen.

For some applications, the duplicate-data-point-identificationfunctionality is configured to determine that, at at least one location,two or more endoluminal data points were acquired by:

sensing a signal that is indicative of the subject's cardiac cycle,while the endoluminal data-acquisition device acquires the plurality ofdata points,

determining that a given data point was acquired at a given phase of thesubject's cardiac cycle, and

in response thereto, identifying the given data point as having beenacquired at a location at which another data point was acquired.

For some applications, the duplicate-data-point-identificationfunctionality configured to determine that the given data point wasacquired at the given phase of the subject's cardiac cycle bydetermining that the given data point was acquired during at least aportion of systole.

For some applications, the duplicate-data-point-identificationfunctionality is configured to determine that, at at least one location,two or more endoluminal data points were acquired by determining thatthe endoluminal data-acquisition device moved past the location in asecond direction with respect to the lumen that is opposite to the firstdirection.

For some applications, the duplicate-data-point-identificationfunctionality configured to determine that the endoluminaldata-acquisition device moved past the location in the second directionwith respect to the lumen by performing image processing on extraluminalimages of the device moving through the lumen generally in the firstdirection.

For some applications, the apparatus further includes a sensorconfigured to detect movement of a portion of the endoluminaldata-acquisition device, and the duplicate-data-point-identificationfunctionality is configured to determine that the endoluminaldata-acquisition device moved past the location in the second directionwith respect to the lumen, in response to a signal from the sensor.

There is further provided, in accordance with some applications of thepresent invention, a method for use with an endoluminal data-acquisitiondevice configured to acquire endoluminal data points while movingthrough a lumen of a subject's body generally in a first direction withrespect to the lumen, including:

while the endoluminal data-acquisition device is being moved through thelumen, acquiring a plurality of endoluminal data points of the lumenusing the endoluminal data-acquisition device;

determining that, while acquiring at least one of the endoluminal datapoints, the endoluminal data-acquisition device was moving in a seconddirection that is opposite to the first direction; and

in response to the determining, generating an output using at least someof the plurality of endoluminal data points of the lumen acquired usingthe endoluminal data-acquisition device.

There is additionally provided, in accordance with some applications ofthe present invention, apparatus for use with an endoluminaldata-acquisition device that acquires a plurality of endoluminal datapoints of a lumen of a body of a subject while being moved through thelumen generally in a first direction with respect to the lumen and adisplay, the apparatus including:

at least one processor including:

-   -   direction-determination functionality configured to determine        that, while acquiring at least one of the endoluminal data        points, the endoluminal data-acquisition device was moving in a        second direction that is opposite to the first direction;    -   output-generation functionality configured, in response to the        determining, to generate an output using at least some of the        plurality of endoluminal data points of the lumen acquired using        the endoluminal data-acquisition device; and    -   display-driving functionality configured to drive the display to        display the output.

For some applications, the direction-determination functionality isconfigured to determine that, while acquiring at least one of theendoluminal data points, the endoluminal data-acquisition device wasmoving in a second direction that is opposite to the first direction byperforming image processing on extraluminal images of the device movingthrough the lumen generally in the first direction.

For some applications, the apparatus further includes a sensorconfigured to detect movement of a portion of the endoluminaldata-acquisition device, and the direction-determination functionalityconfigured to determine that, while acquiring at least one of theendoluminal data points, the endoluminal data-acquisition device wasmoving in a second direction that is opposite to the first direction, inresponse to a signal from the sensor.

For some applications,

the endoluminal data-acquisition device includes an endoluminal imagingprobe configured to acquire a plurality of endoluminal image frames ofthe lumen, and

the output-generation functionality is configured to generate the outputby generating an endoluminal image stack using at least some of theplurality of endoluminal image frames of the lumen.

For some applications, the output-generation functionality is configuredto generate the output by placing image frames in order within the imagestack based on determining that, while acquiring at least one of theendoluminal data points, the endoluminal data-acquisition device wasmoving in the second direction.

For some applications, the output-generation functionality is configuredto generate the output by generating an indication on the endoluminalimage stack of at least a portion of the endoluminal image stack thatwas acquired by the data-acquisition device while the data-acquisitiondevice was moving in the second direction.

For some applications, the direction-determination functionality isconfigured to determine that, while acquiring at least one of theendoluminal data points, the endoluminal data-acquisition device wasmoving in a second direction that is opposite to the first direction by:

sensing a signal that is indicative of the subject's cardiac cycle,while the endoluminal data-acquisition device acquires the plurality ofdata points,

determining that a given data point was acquired at a given phase of thesubject's cardiac cycle, and

in response thereto, identifying the given data point as having beenacquired while the endoluminal data-acquisition device was moving in thesecond direction.

For some applications, the direction-determination functionality isconfigured to determine that the given data point was acquired at thegiven phase of the subject's cardiac cycle by determining that the givendata point was acquired during at least a portion of systole.

The present invention will be more fully understood from the followingdetailed description of embodiments thereof, taken together with thedrawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart, at least some of the steps of which are used inprocedures that utilize co-use of endoluminal data and extraluminalimaging, in accordance with some applications of the present invention;

FIGS. 2A-B are schematic illustrations of an endoluminal device beinginserted into a lumen, and (in FIG. 2B) a sensor for sensing thedistance traveled through the lumen by the endoluminal device relativeto a known starting location, in accordance with some applications ofthe present invention;

FIG. 3 is a flow chart, at least some of the steps of which are used inprocedures that utilize co-use of endoluminal data and extraluminalimaging, in accordance with some applications of the present invention;

FIG. 4 shows an initial best angiogram of a luminal segment, the initialbest angiogram being generated prior to the commencement of the pullbackof an endoluminal imaging probe, in accordance with some applications ofthe present invention;

FIG. 5 shows a post-pullback best angiogram of a luminal segment, thepost-pullback best angiogram being generated subsequent to thetermination of the pullback of an endoluminal imaging probe, inaccordance with some applications of the present invention;

FIG. 6 shows a combined best angiogram of a luminal segment, thecombined best angiogram being generated by co-registering the initialbest angiogram and the post-pullback best angiogram, in accordance withsome applications of the present invention;

FIG. 7 shows the co-use of previously-acquired endoluminal images and anextraluminal fluoroscopic image, in accordance with some applications ofthe present invention;

FIG. 8 shows a location on an extraluminal image of a lumen that hasbeen selected, the index of the corresponding endoluminal image framebeing derived in response thereto, in accordance with some applicationsof the present invention;

FIG. 9 shows co-display of previously-acquired endoluminal images and anextraluminal fluoroscopic image, in accordance with some applications ofthe present invention;

FIG. 10 shows the co-use of previously-acquired endoluminal images and acurrent extraluminal fluoroscopic image stream, in accordance with someapplications of the present invention;

FIG. 11 shows the co-use of a stack of previously-acquired IVUS imagesand a current, extraluminal fluoroscopic image stream, in accordancewith some applications of the present invention;

FIG. 12 is a graph indicating typical movement of an endoluminal imagingprobe during pullback of the probe; and

FIG. 13 shows an extraluminal image of a stent inside a blood vesselthat has been enhanced, in accordance with some applications of thepresent invention.

DETAILED DESCRIPTION OF EMBODIMENTS

-   -   The terms “medical tool,” “tool”, “device,” and “probe” refer to        any type of a diagnostic or therapeutic or other functional tool        including, but not limited to, a cardiovascular catheter, a        stent delivery and/or placement and/or retrieval tool, a balloon        delivery and/or placement and/or retrieval tool, a valve        delivery and/or repair and/or placement and/or retrieval tool, a        graft delivery and/or placement and/or retrieval tool, a tool        for the delivery and/or placement and/or retrieval of an        implantable device or of parts of such device, an implantable        device or parts thereof, a tool for closing a gap, a tool for        closing a septal defect, a guide wire, a marker wire, a suturing        tool, a clipping tool (such as a valve-leaflet-clipping tool), a        biopsy tool, an aspiration tool, a navigational tool, a        localization tool, a probe comprising one or more location        sensors, a tissue characterization probe, a probe for the        analysis of fluid, a measurement probe, an electrophysiological        probe, a stimulation probe, an ablation tool, a tool for        penetrating or opening partial or total occlusions in blood        vessels, a drug or substance delivery tool, a chemotherapy tool,        a photodynamic therapy tool, a brachytherapy tool, a local        irradiation tool, a laser device, a tool for delivering energy,        a tool for delivering markers or biomarkers, a tool for        delivering biological glue, an irrigation device, a suction        device, a ventilation device, a device for delivering and/or        placing and/or retrieving a lead of an electrophysiological        device, a lead of an electrophysiological device, a pacing        device, a coronary sinus device, an imaging device, a sensing        probe, a probe comprising an optical fiber, a robotic tool, a        tool that is controlled remotely, an excision tool, a plaque        excision tool (such as a plaque excision catheter) or any        combination thereof    -   The terms “image” and “imaging” refer to any type of medical        imaging, typically presented as a sequence of images and        including, but not limited to, imaging using ionizing radiation,        imaging using non-ionizing radiation, video, fluoroscopy,        angiography, ultrasound, CT, MR, PET, PET-CT, CT angiography,        SPECT, Gamma camera imaging, Optical Coherence Tomography (OCT),        Near-Infra-Red Spectroscopy (NIRS), Vibration    -   Response Imaging (VRI), Optical Imaging, infrared imaging,        electrical mapping imaging, other forms of Functional Imaging,        or any combination or fusion thereof. Examples of ultrasound        imaging include Endo-Bronchial Ultrasound (EBUS), Trans-Thoracic        Echo (TTE), Trans-Esophageal Echo (TEE), Intra-Vascular        Ultrasound (IVUS), Intra-Cardiac Ultrasound (ICE), or any        combination thereof    -   The term “contrast agent,” when used in reference to its        application in conjunction with imaging, refers to any substance        that is used to highlight, and/or enhance in another manner, the        anatomical structure, functioning, and/or composition of a        bodily organ while the organ is being imaged.    -   The term “stabilized,” when used in the context of displayed        images, means a display of a series of images in a manner such        that periodic, cyclical, and/or other motion of the body        organ(s) being imaged, and/or of a medical tool being observed,        is partially or fully reduced, with respect to the entire image        frame, or at least a portion thereof    -   The term “automatic,” when used for describing the generation        and utilization of the road map, means “without necessitating        user intervention or interaction.” (Such interaction or        intervention may still however be optional in some cases.)    -   The term “real time” means without a noticeable delay.    -   The term “near real time” means with a short noticeable delay        (such as approximately one or two motion cycles of the        applicable organ, and, in the case of procedures relating to        organs or vessels the motion of which are primarily as a result        of the cardiac cycle, less than two seconds).    -   The term “on-line,” when used in reference to image processing,        or to measurements being made on images, means that the image        processing is performed, and/or the measurements are made,        intra-procedurally, in real time or near real time.

Applications of the present invention are typically used during medicalprocedures that are performed, in whole or in part, on or within luminalstructures. For some applications, apparatus and methods provided hereinfacilitate the co-use of extraluminal imaging and endoluminal data inperforming such medical procedures. Endoluminal data may include imagingdata, data derived from measurements, other data, or any combinationthereof.

For some applications, the co-use of the endoluminal data and theextraluminal images is performed in the following manner. Endoluminaldata are acquired by positioning an endoluminal data-acquisition devicealong a luminal segment of interest that includes a designated luminalsite. Subsequently, while observing extraluminal images of the luminalsegment, one or more locations along that segment are indicated by auser input device. In response to the indication of the one or morelocations by the user input device, the corresponding,previously-acquired endoluminal images are displayed.

Typically, the designated luminal site includes a site being diagnosed,and at which, subject to the outcome of the diagnosis, a therapeuticdevice will be positioned and deployed, e.g., the site of an anatomicalfeature, the implantation site of a previously-implanted device, and/ora site at a defined location with respect to the implantation site. Forexample, the designated luminal site may include a portion of the lumenthat is narrow with respect to surrounding portions of the lumen, and/orthe site of a lesion.

For some applications, the co-use of the endoluminal data and theextraluminal images is performed in the following manner. Endoluminaldata are acquired by positioning an endoluminal data-acquisition deviceat a designated luminal site. Subsequently, an endoluminal therapeuticdevice is positioned and deployed at the designated luminal site underextraluminal imaging, while concurrently viewing on-line the endoluminaldata that were previously acquired by the endoluminal data-acquisitiondevice at the current location of the therapeutic device. Typically,endoluminal data are acquired at respective endoluminal sites in thevicinity of the designated endoluminal site. Subsequently, when theendoluminal therapeutic device is placed inside the lumen,previously-acquired endoluminal data are displayed and updated,typically automatically and typically on-line, to correspond to thecurrent location of the therapeutic device (or of a portion thereof),the location of the therapeutic device typically changing during thepositioning of the therapeutic device.

For some applications, extraluminal imaging and the previously-acquiredendoluminal data are co-used such that it is as if the therapeuticdevice is being positioned and deployed under both real-timeextraluminal imaging and real-time endoluminal data acquisition. This isbecause (a) the extraluminal imaging is performed in real-time, and (b),although the endoluminal data are not acquired in real-time, endoluminaldata are displayed that correspond to the current location of thetherapeutic device.

In accordance with some applications of the present invention, when thetherapeutic device is disposed inside the lumen, the location of thedevice within the lumen is determined by performing image processing onthe extraluminal image of the device inside the lumen.

For some applications, the image processing includes tracking of one ormore visible portions of a moving therapy-applying portion of the devicein the extraluminal images. Typically, the tracking is performed in realtime, and, typically, in accordance with techniques described in US2010/0228076 to Blank, which is incorporated herein by reference.

For some applications, the image processing includes stabilization of animage stream produced by the extraluminal imaging. Typically, thestabilization is performed in real time, and typically in accordancewith techniques described in US 2008/0221442 to Tolkowsky, or US2010/0228076 to Blank, both of which applications are incorporatedherein by reference. Typically, the stabilization facilitates the co-useof the endoluminal data with the extraluminal images (particularly incases of intense organ motion). This is because it is typically easierto determine the luminal location of the therapeutic device based upon astabilized image stream than to determine the luminal location of thetherapeutic device on a native, non-stabilized image stream.

For some applications, the stabilized image stream is also enhanced,typically in real time, typically in accordance with techniquesdescribed in US 2010/0228076 to Blank.

For some applications, during the acquisition of the endoluminal data bythe endoluminal data-acquisition device, the location of the endoluminaldata-acquisition device is determined by advancing the endoluminaldata-acquisition device under extraluminal imaging and image processingthe extraluminal images to determine the location of a movingdata-acquiring portion of the endoluminal data-acquisition device. Forsome applications, during this stage, the extraluminal image stream isstabilized and/or enhanced, as described hereinabove, to facilitate thedetermination of the location of the endoluminal data-acquisitiondevice, based upon the extraluminal images. Alternatively, othertechniques are used for determining the location of the endoluminaldata-acquisition device, as described hereinbelow.

For some applications, the luminal structure to which the apparatus andmethods described herein are applied includes a lumen in the vascularsystem, the respiratory tract, the digestive tract, the urinary tract,or any other luminal structure within a patient's body.

For some applications, the endoluminal data-acquisition device is animaging probe. For some applications, the imaging probe is an IVUSprobe, an EBUS probe, another ultrasound probe, an OCT probe, an NIRSprobe, an MR probe, or any combination thereof.

For some applications, the endoluminal data-acquisition device performsadditional functions. For example, the endoluminal data-acquisitiondevice may comprise a probe, such as the VIBE™ RX Vascular ImagingBalloon Catheter, marketed by Volcano Corporation (San Diego, USA), thatincludes both IVUS and coronary balloon functionalities.

For some applications, the endoluminal data-acquisition device acquiresdata in a form other than images. For example, the data may include datarelated to pressure, flow, temperature, electrical activity, or anycombination thereof. For some applications, and typically when data areacquired with respect to a coronary vessel, the endoluminaldata-acquisition device is a Fractional Flow Reserve (FFR) probe.

For some applications, the extraluminal imaging is fluoroscopy, CT, MR,PET, SPECT, ultrasound, or any combination thereof.

For some applications, the apparatus and methods described herein areused with a therapeutic device that is positioned and/or deployed at ananatomical feature that requires or potentially requires treatment, suchas a partial or total occlusion, a native valve, an aneurism, adissection, a malformation, a septal defect, a mass suspected of beingmalignant, a mass suspected of being inflammatory, etc. The endoluminaldata are typically determined at, and/or in the vicinity of, theanatomical feature.

For some applications, apparatus and methods described herein are usedwith a therapeutic device that is positioned and/or deployed at animplantation site of a previously-implanted device such as a stent, agraft or a replacement valve. The endoluminal data are determined at,and/or in the vicinity of, the implantation site. For example, thetechniques described herein may be used during the placement of a newprosthetic aortic valve at the site of (e.g., inside) a previouslyimplanted prosthetic aortic valve that is no longer functioning.

For some applications, apparatus and methods described herein are usedwith a therapeutic device that is positioned and/or deployed at adefined location relative to a previously-implanted device such as astent, a graft or a replacement valve. The endoluminal data aredetermined at and in the vicinity of the defined location. For example,the techniques described herein may be used during the placement of acoronary stent such that the new stent overlaps with or is adjacent to apreviously-implanted stent, in order to treat a long lesion and/or alesion that has diffused along a coronary artery.

Reference is now made to FIG. 1, which is a flow chart, at least some ofthe steps of which are used in the course of co-use of endoluminal dataand extraluminal imaging, in accordance with some applications of thecurrent invention. It is noted that, for some applications, some of thesteps shown in FIG. 1 may be practiced, without all of the steps shownin FIG. 1 necessarily being practiced in combination.

In phase 1, extraluminal imaging is activated. Typically theextraluminal imaging is activated at this stage, in order to facilitatedetermination of the location of a moving data-acquiring portion theendoluminal data-acquisition device by performing image processing onthe extraluminal images, and/or in order to facilitate the insertion ofthe endoluminal data-acquisition device. For some applications, methodsother than extraluminal imaging are used for determining the location ofthe endoluminal data-acquisition device, for example, as describedhereinbelow. For some applications (e.g., during insertion of thedata-acquisition device into an endobronchial lumen, which may beperformed without the guidance of extraluminal imaging), theextraluminal imaging is not activated at this stage.

In phase 2, the extraluminal image stream is typically stabilized, andoptionally enhanced, typically in accordance with techniques previouslydisclosed in US 2008/0221442 to Tolkowsky, and/or US 2010/0228076 toBlank, both of which applications are incorporated herein by reference.For some applications, the extraluminal image stream is stabilized withrespect to radiopaque markers on the endoluminal data-acquisitiondevice.

In phase 3, the endoluminal data-acquisition device is inserted towardsthe designated site. The designated site is typically a site beingdiagnosed, and at which, subject to the outcome of such diagnosis, thetherapeutic device will be positioned and deployed, e.g., the site of ananatomical feature, the implantation site of a previously-implanteddevice, and/or a site at a defined location with respect to theimplantation site, as described hereinabove. The endoluminaldata-acquisition device is typically imaged by extraluminal imaging.

In phase 4, endoluminal data, typically images, are acquired by theendoluminal data-acquisition device. Typically, data are acquired atand/or in the vicinity of the designated site. Typically, a plurality ofdata points (e.g., images) are acquired at respective locations alongthe lumen. It is noted that, for some applications, data are acquiredsubsequent to the initial insertion of the data-acquisition device intothe lumen. For example, when data are acquired from blood vessels, thedata-acquisition device is typically inserted into the blood vessel tobeyond the site of interest under extraluminal imaging (e.g.,fluoroscopy). Data acquisition is typically performed during (manual orautomated) pullback of the data-acquisition device through the bloodvessel. In alternative applications, e.g., when data are acquired froman endobronchial airway, data are typically acquired by thedata-acquisition device during insertion of the data-acquisition deviceinto the airway.

For some applications, in the course of pullback of the data-acquisitiondevice, the lumen (for example, a coronary artery) also experiences acyclical motion (for example, due to the cardiac cycle) that causes itto pulsate and move back and forth relatively to the endoluminaldata-acquisition device. For some applications, e.g., in the case of alumen that undergoes such back-and-forth cyclical motion, data acquiredby the endoluminal data-acquisition device are gated to the cyclicalmotion cycle of the lumen. Subsequently, endoluminal data acquired inthe course of the pullback at at-least-one specific phase of the motioncycle of the lumen are co-registered with one or more extraluminalimages acquired, and gated, at the corresponding at-least-one phaseduring the pullback, in order to facilitate co-registration of theendoluminal data with the extraluminal images, in accordance with thetechniques described herein. For some applications, co-registeringendoluminal data with extraluminal images that are gated to the samephase as the phase to which the endoluminal data were gated, reducesdistortions in the co-registration that may be introduced due to thecyclical motion of the lumen in the absence of using the aformentionedgating techniques.

For some applications, there is a single, gated extraluminal angiogramimage to which all gated endoluminal data are co-registered. For someapplications, a three-dimensional model is generated from two (or more)two-dimensional gated angiograms, and the gated endoluminal data isco-registered with that three-dimensional model.

For some applications, the commencement and/or termination of pullbackare identified, typically automatically and typically on-line, by meansof image processing. For some applications, the image processing isperformed by an image comparator which identifies a change (such as inthe color of image pixels or in the geometry of image features) in thesequentially-acquired endoluminal images, and interprets the change asindicating the commencement of pullback. For some applications, theimage processing is performed by an image comparator which identifies adiminishing change in the sequentially-acquired endoluminal images, andinterprets the diminishing change as indicating the termination ofpullback.

For some applications, the commencement and/or termination of pullbackare identified by means of a signal transmitted by the pullback unitand/or by the endoluminal data acquisition system. For someapplications, the commencement and/or termination of pullback areindicated by means of user input.

In phase 5, each applicable image or data point acquired in phase 4 is,typically automatically, assigned a location. The locations assigned torespective data points (e.g., images) correspond to the location of theendoluminal data-acquisition device when the respective data points areacquired. Typically, this step is performed simultaneously with phase 4,such that the system assigns locations corresponding to respective datapoints at the time of the acquisition of the data points.

For some applications, the location of a data-acquiring portion of theendoluminal data-acquisition device that moves during pullback, and aportion of which is visible in the extraluminal imaging, is identifiedvia image processing. For example, radiopaque markers on a movingimaging portion of the endoluminal data-acquisition device may beidentified in extraluminal fluoroscopic images. For some applications,the visible portion is identified and tracked, typically on-line andtypically automatically, for example, in accordance with techniquesdescribed in US 2010/0228076 to Blank.

For some applications, the location of the moving, visible portion ofthe endoluminal data-acquisition device is determined relative to ananatomical feature visible in the extraluminal imaging. For someapplications, the feature is a bifurcation, a curve or some other uniqueshape, a partial or total occlusion, a native valve, an aneurism, aseptal defect, or a malformation. For some applications, contrast agentis injected in order to make the feature visible (for example, in thecase of vasculature that is imaged under fluoroscopy). Typically, thequantity and concentration of the contrast agent that is injected issuch that, in some image frames, both the visible portion of theendoluminal data-acquisition device and the anatomical feature may bediscerned concurrently in the extraluminal image.

For some applications, the location of the moving, visible portion ofthe endoluminal data-acquisition device is determined relative to apreviously-deployed device visible in the extraluminal imaging. For someapplications, the previously-deployed device is a stent, or a graft, ora replacement valve.

For some applications, the location of the moving, visible portion ofthe endoluminal data-acquisition device is determined relative tovisible markers along a guide wire along which the endoluminaldata-acquisition device is inserted.

For some applications, the location of the moving, visible portion ofthe endoluminal data-acquisition device is determined according to itsdistance along a guide wire along which the endoluminal data-acquisitiondevice is inserted, the distance typically being measured relative tothe distal tip of a guiding catheter through which the guide wire waspreviously inserted (or relative to any other of the aforementionedvisible features). For some applications, the endoluminaldata-acquisition device includes a portion that substantially does notmove with respect to the lumen during pullback, such as an insertionsheath. The location of moving, visible portion of the data-acquisitiondevice is determined, via image processing, with reference to theportion of the device that substantially does not move with respect tothe lumen during pullback.

For some applications, the location of the moving visible portion of theendoluminal data-acquisition device is determined by means of displaycoordinates. Typically, for such applications, when an endoluminaltherapeutic device is subsequently inserted into the lumen in order totreat the designated site, the same viewing angle of the extraluminalimaging device relative to the lumen, and the same zoom level of theextraluminal imaging are used as were used to image the endoluminaldata-acquisition device inside the lumen. For such applications, theposition of the subject typically remains substantially unchangedbetween the insertion of the data-acquisition device and the insertionof the therapeutic device. Thus, the location of the endoluminaldata-acquisition device within the lumen may be matched with thelocation of the therapeutic device that is subsequently inserted intothe lumen.

For some applications, the location of the moving, visible portion ofthe endoluminal data-acquisition device is determined by determining adistance traveled by the device along the lumen, from a known startinglocation. For some applications, the distance is measured by a pullbackunit to which the device is connected. For some applications, thedistance is measured by a longitudinal position/movement sensor coupledto apparatus through which the endoluminal data-acquisition device isinserted, e.g., as described hereinbelow with reference to FIG. 2. Forsome applications, the apparatus is a guiding catheter. Typically, thesensor measures the extent of longitudinal movement (e.g., insertion,pullback) of a proximal portion of the device. For some applications,the sensor is optical (e.g., laser-based), or mechanical, or electric,or magnetic, or any combination thereof. For some applications, inresponse to measuring the extent of the longitudinal motion of theproximal portion of the device, the system estimates a distance by whichthe moving, data-acquiring portion has moved along the lumen (typically,along a center line of the lumen), typically automatically and typicallyon-line. The center line is determined, typically automatically, inaccordance with techniques described in US 2010/0228076 to Blank, whichis incorporated herein by reference.

For some applications, the location of the moving portion of theendoluminal data-acquisition device is determined according totechniques described in US Patent Application 2006/0241465 and US PatentApplication 2007/0038061, both to Huennekens, and both of whichapplications are incorporated herein by reference. For someapplications, techniques as described in U.S. Pat. No. 5,357,550 toAsahina, US 2011/0034801 to Baumgart, and/or U.S. Pat. No. 7,729,746 toRedel are applied, in order to determine the location of the movingportion of the endoluminal data-acquisition device. All of theaforementioned references are incorporated herein by reference.

For some applications, the location of the endoluminal data-acquisitiondevice is determined even in the absence of simultaneous extraluminalimaging. For example, it may be determined that the device is at ananatomical feature such as a bifurcation, based upon the images or dataacquired by the device. Subsequently, the device may be pulled back at aknown speed, by a pullback unit to which the device is connected.Alternatively, the distance by which the device has been pulled back atthe acquisition of respective data points may be measured. Thus, it maybe determined, at the time of acquisition of a given image or a givendata point, what is the location of the device relative to theanatomical feature. Separately (before or after acquisition of theendoluminal data), the anatomical feature is identified in anextraluminal image of the lumen. Based upon the location of theanatomical feature in the extraluminal image, endoluminal data points(e.g., images) are assigned to respective locations within theextraluminal image.

For some applications, other techniques are applied, e.g., techniquesdescribed hereinbelow with reference to FIG. 3 are applied, in order todetermine the location of the endoluminal data-acquisition device whenthe respective data points (e.g., images) are acquired.

In phase 6, the endoluminal data-acquisition device is typicallyretrieved from the designated site (and, further typically, withdrawnfrom the lumen), in order to accommodate the insertion of an endoluminaldevice (e.g., an endoluminal therapeutic device) into the lumen.

In phase 7, while observing extraluminal images of the luminal segmentcomprising the designated location, one or more locations along thatsection are indicated by a user input device. In response thereto, thepreviously-acquired endoluminal images corresponding to the one or morelocations are displayed. For some applications, the user input device isused to select the one or more locations. Typically, the user designatesa location using the user input device, and, in response thereto,typically automatically and on-line, the system identifies a locationalong the lumen (e.g., along the luminal center line) as correspondingto the designated location, and retrieves and displays a correspondingendoluminal image. For some applications, the center line is generatedin accordance with techniques described in US 2010/0220917 to Steinberg,which is incorporated herein by reference.

Alternatively or additionally, by observing an angiogram frame side byside with endoluminal image frames of the luminal segment comprising thedesignated location, one or more locations along the section areindicated by a user input device with respect to endoluminal imagingdata. For some applications, the user indication is made upon theendoluminal image stack. For some applications, the user indication ismade by browsing through the endoluminal images. In response toreceiving the user indication, the location along the lumen (e.g., alongthe luminal center line) within the angiogram corresponding to thelocation indicated with respect to an endoluminal image or theendoluminal image stack is determined and indicated.

Typically, a clinical diagnosis is facilitated by an operator viewingpreviously-acquired endoluminal images corresponding to the one or morelocations selected on extraluminal images of the luminal segment, or bythe operator viewing indications of locations on an extraluminal imagethat correspond to one or more locations selected with respect toendoluminal images or an endoluminal image stack, as described withreference to phase 7. Alternatively, a clinical diagnosis is made by theoperator reviewing the extraluminal images and/or the endoluminal data(and/or by reviewing other data), without performing phase 7. Typically,a therapeutic process, such as the one described in phase 8 and beyond,is performed based upon the clinical diagnosis made by the operator.

In phase 8, an endoluminal therapeutic device is inserted to thedesignated location under extraluminal imaging. Typically, stabilization(and optionally also enhancement) is applied, typically on-line andtypically automatically, to the extraluminal image stream, in agenerally similar manner to that described with reference to phase 2. Atleast a portion of a therapy-applying portion of the endoluminaltherapeutic device, or a probe used for the insertion of the therapeuticdevice (i.e., an insertion probe) that moves with respect to the lumen,is typically visible in the extraluminal images. For example, thetherapy-applying portion may include radiopaque markers, forapplications in which the extraluminal imaging is performed viafluoroscopy.

In phase 9, the current location of the moving, visible portion of theendoluminal therapeutic device or the insertion probe is determined,typically on-line and typically automatically. Typically, the currentlocation of the device is determined while the device is at or in thevicinity of the designated site. Typically, phase 9 is performedsimultaneously with phase 8, i.e., while the endoluminal therapeuticdevice is at respective current locations, the current location of thedevice is determined by the system.

For some applications, the location of the portion of the endoluminaltherapeutic device or of the insertion probe that is visible in theextraluminal imaging is identified via image processing. For example,radiopaque markers on the endoluminal therapeutic device may beidentified in extraluminal fluoroscopic images. For some applications,the visible portion is identified and tracked, typically on-line andtypically automatically, for example, in accordance with techniquesdescribed in US 2010/0228076 to Blank.

For some applications, the location of the moving, visible portion ofthe endoluminal therapeutic device or the insertion probe is determinedrelative to an anatomical feature visible in the extraluminal imaging.For some applications, the feature is a bifurcation, a curve or someother unique shape, a partial or total occlusion, a native valve, ananeurism, a septal defect, or a malformation. For some applications,contrast agent is injected, in order to make the feature visible (forexample, in the case of vasculature that is imaged under fluoroscopy).Typically, the quantity and concentration of the contrast agent that isinjected is such that, in some image frames, both the visible portion ofthe endoluminal data-acquisition device and the anatomical feature maybe discerned concurrently in the extraluminal image.

For some applications, the location of the moving, visible portion ofthe endoluminal therapeutic device or the insertion probe is determinedrelative to a previously-deployed device visible in the extraluminalimaging. For some applications, the device is a stent, or a graft, or areplacement valve.

For some applications, the location of the moving, visible portion ofthe endoluminal therapeutic device or the insertion probe is determinedrelative to visible markers along a guide wire along which the device isinserted.

For some applications, the location of the moving, visible portion ofthe endoluminal therapeutic device or the insertion probe is determinedaccording to its distance along a guide wire along which the deviceand/or the probe is inserted, the distance typically being measuredrelative to the distal tip of a guiding catheter through which the guidewire was previously inserted. For some applications, the endoluminaltherapeutic device includes a portion that substantially does not movewith respect to the lumen during a stage of the advancement of thetherapy-applying portion of the device, such as an insertion sheath. Thelocation of moving, visible portion of the endoluminal therapeuticdevice is determined, via image processing, with reference to theportion of the device that substantially does not move with respect tothe lumen.

For some applications, the location of the moving, visible portion ofthe endoluminal therapeutic device or the insertion probe is determinedby determining a distance traveled by the device along the lumen, from aknown starting location. For some applications, such a distance ismeasured by a pullback unit to which the device is connected. For someapplications, the distance is measured by a longitudinalposition/movement sensor coupled to an apparatus through which theendoluminal data-acquisition device is inserted, e.g., as describedhereinbelow with reference to FIG. 2. For some applications, theapparatus is a guiding catheter. Typically, the sensor measures theextent of longitudinal movement (e.g., insertion, pullback) of aproximal portion of the device. For some applications, the sensor isoptical (e.g., laser-based), or mechanical, or electric, or magnetic, orany combination thereof. For some applications, in response to measuringthe extent of the longitudinal motion of the proximal portion of thedevice, the system estimates a distance by which the therapy-applyingportion of the device has moved along the lumen (e.g., along a centerline of the lumen), typically automatically and typically on-line. Thecenter line is determined, typically automatically, in accordance withtechniques described in US 2010/0228076 to Blank.

For some applications, the location of the moving, visible portion ofthe therapeutic device is determined by means of display coordinates.Typically, as described hereinabove, the current location of thetherapeutic device may be matched with a location of the endoluminaldata-acquisition device by using the same viewing angle of theextraluminal imaging device relative to the lumen, and by using zoomlevel of the extraluminal imaging, as were used for the extraluminalimaging of the endoluminal data-acquisition device.

In phase 10, data points (e.g., images) that were previously acquired bythe endoluminal data-acquisition device at or near the location areretrieved and associated, typically on-line and typically automatically,with the extraluminal imaging, while the device is at or near the samelocation.

In phase 11, data points (e.g., images) that were previously acquired bythe endoluminal data-acquisition device at or near the location aredisplayed together with the extraluminal imaging. Typically, data pointsare displayed that correspond to the current location of the endoluminaltherapeutic device (as determined in phase 9). Typically, phases 10 and11 are performed in real time with respect to phases 8 and 9. Thus,while the endoluminal therapeutic device is at respective currentlocations inside the lumen, the location of the device is determined,and the endoluminal data points associated with the location areretrieved and displayed.

For some applications, endoluminal and extraluminal images correspondingto the same location (typically, the current location of the endoluminaltherapeutic device) are displayed side by side. For some applications,endoluminal and extraluminal images corresponding to the same location(typically, the current location of the endoluminal therapeutic device)are merged, such as by means of fusion or overlay. For someapplications, quantitative vessel analysis (QVA) data are displayed, thedata typically corresponding to the current location of the endoluminaltherapeutic device. Typically, the QVA data are generated automaticallyand on-line in accordance with techniques described in US 2010/0228076to Blank, which is incorporated herein by reference. For example, thecurrent location of one or more markers of the therapeutic device may bedetermined via image-processing, and QVA data corresponding to thecurrent location of the markers may be generated and displayed,typically automatically, and typically on-line. Alternatively, inresponse to a location within the lumen being indicated via a user inputdevice, QVA data corresponding to the location may be generated anddisplayed, typically automatically, and typically on-line.

For some applications, enhanced extraluminal images of a lumen segmentcomprising the location are generated, for example, in accordance withtechniques described in US 2010/0228076 to Blank, which is incorporatedherein by reference.

For some applications (for example, in applications in which theendoluminal data-acquisition device is an ultrasound probe), imageslices corresponding to a luminal segment at or around the designatedsite are displayed as stacked.

Typically, the effect of co-displaying the endoluminal data with theextraluminal imaging is as if the endoluminal therapeutic device isbeing positioned and deployed under real-time extraluminal imaging andusing real-time endoluminal data acquisition, at and in the vicinity ofthe designated site.

For some applications, phases 1 through 7 (or any applicable subset ofthose phases) are repeated subsequent to the deployment of thetherapeutic device, such as in the course of performing a clinicalevaluation of the outcome of the deployment of that device. For example,phases 1-7 may be repeated so as to facilitate the co-display ofendoluminal images of the lumen, post-deployment of the device, with oneor more extraluminal images of the lumen.

For some applications, a procedure is carried out generally inaccordance with the flowchart shown in FIG. 1, in which one, some, orall of the following apply:

-   -   The luminal structure is a coronary artery.    -   The designated site for diagnosis and treatment is a        partially-occluded segment of the artery.    -   The endoluminal data-acquisition device is an IVUS probe,        capable of identifying coronary disease in the luminal wall.    -   The extraluminal imaging is performed via fluoroscopy.    -   The fluoroscopic image stream is, for some applications,        stabilized on-line.    -   The endoluminal therapeutic device that is positioned and        deployed at the site of the occlusion is a balloon-expandable        stent.    -   The balloon carrying the stent comprises radio-opaque markers at        its proximal and distal ends, the markers being visible under        fluoroscopic imaging.

For some such applications, images generated by the IVUS probe withinthe coronary vessel are used in conjunction with the extraluminalfluoroscopic image stream in the following manner:

i. An IVUS catheter is inserted to the site of an occlusion underfluoroscopic imaging, to inspect endoluminal anatomy.

ii. Optionally, the fluoroscopic image stream is stabilized. For someapplications, the image stream is stabilized with respect to radiopaquesegments of the IVUS catheter.

iii. The image slices generated by the IVUS are recorded and stored intandem with the visual location (such as display coordinates) of thedistal tip of the IVUS catheter as seen by the image-stabilized streamof the fluoroscopy.

iv. The IVUS catheter is retrieved to make room for balloon/stentdeployment.

v. A catheter with a balloon and/or stent is inserted to the site of theocclusion, under fluoroscopic imaging.

vi. The location of the distal tip of the catheter carrying the balloonand/or stent is visually recognized (such as via display coordinates).

vii. The IVUS images previously recorded at the same location aredisplayed, together with the fluoroscopic images. For some applications,the IVUS images are displayed in a separate window (but on the samescreen as the fluoroscopic images). For some applications, the IVUSimages are displayed on a separate screen. For some applications, theIVUS images being displayed are two-dimensional (also known as“slices”). For some applications, a stack comprising multiple slices isdisplayed. For some applications, a three-dimensional “tunnel-like”reconstruction of the IVUS images of the vessel (or a section thereof)is displayed. For some applications, the IVUS images are overlaid on thefluoroscopic images. For some applications, the IVUS images are fusedwith the fluoroscopic images.

viii. As a result, the balloon and/or stent may be positioned anddeployed based upon an on-line combination of real-time fluoroscopicimages and of IVUS images recorded earlier (for example, several minutesearlier).

For some applications, data acquired by a first endoluminal modality(e.g., IVUS) are co-registered with the fluoroscopic image stream, inaccordance with the applications described hereinabove. Subsequently,data acquired by a second endoluminal modality (e.g., OCT) areco-registered with the fluoroscopic image stream, in accordance with theapplications described hereinabove. Consequently, due to both data setsbeing co-registered with the fluoroscopic image stream, the two datasets are co-registered to one another. For some applications, the twoendoluminal data sets are displayed as overlaid or otherwise merged withone another.

For some applications, generally similar steps to those described withreference to FIG. 1 are performed, except for the following differences.In phase 8, instead of a therapeutic endoluminal device being insertedinto the lumen, a second endoluminal data-acquisition device is insertedinto the lumen. Typically, the first and second endoluminaldata-acquisition devices acquire endoluminal images using respectiveimaging modalities. For example, in phase 3, an IVUS probe may beinserted into the lumen, and in phase 8 an OCT probe may be insertedinto the lumen, or vice versa.

In phase 9, the current location of the second endoluminaldata-acquisition device is determined, for example, using any of thetechniques described herein (such as, by performing image processing onextraluminal images of the second endoluminal data-acquisition deviceinside the lumen). In phases 10 and 11, endoluminal images which werepreviously acquired using the first data-acquisition device at thecurrent location of the second endoluminal data-acquisition device areretrieved and displayed, typically on-line and typically automatically.

Typically, the endoluminal images which were acquired using the firstdata-acquisition device at the current location of the secondendoluminal data-acquisition device are displayed together withendoluminal images that are being acquired in real time by the secondendoluminal data-acquisition device, while the second endoluminaldata-acquisition device is at the current location. For someapplications, endoluminal images that are acquired in real time by thesecond endoluminal data-acquisition device, while the second endoluminaldata-acquisition device is at the current location, are displayedtogether with an indication of the current location of the secondendoluminal data-acquisition device with respect to an endoluminal imagestack generated using endoluminal images that were previously acquiredby the first endoluminal data-acquisition device. For some applications,using the above-described technique, data acquired by first and secondendoluminal data acquisition devices are registered with respect to oneanother, and the co-registered data are displayed subsequent totermination of the acquisition of endoluminal images by both the firstand the second endoluminal data-acquisition devices. For someapplications, endoluminal images corresponding to the current locationof the second endoluminal data-acquisition device that were acquired bythe first endoluminal data acquisition device, and/or by the secondendoluminal data acquisition device, are co-displayed with an indicationof the current location of the second endoluminal data-acquisitiondevice on an extraluminal image of the lumen, using the techniquesdescribed herein.

For some applications, techniques described herein (e.g., techniquesdescribed with reference to FIGS. 1-11) are performed by a system thatincludes at least one processor, for use with an endoluminaldata-acquisition device that is configured to acquire a set ofendoluminal data-points with respect to a lumen of a body of a subjectat respective locations inside the lumen, and a second endoluminaldevice. The processor typically includes (a) location-associationfunctionality configured to associate a given endoluminal data pointacquired by the endoluminal data-acquisition device with a givenlocation within the lumen, (b) location-determination functionalityconfigured, in an extraluminal image of the second endoluminal device,to determine by means of image processing, a current location of atleast a portion of the second endoluminal device inside the lumen, and(c) display-driving functionality configured, in response to determiningthat the second endoluminal device is currently at the given location,to drive a display to display an indication of the endoluminal datapoint associated with the given location.

Reference is now made to FIG. 2A-B, which are schematic illustrations ofan endoluminal device 31 (e.g., an IVUS probe) being inserted into alumen, and (in FIG. 2B) a sensor 36 for sensing the distance traveledthrough the lumen by the endoluminal device relative to a known startinglocation, in accordance with some applications of the present invention.FIG. 2A shows IVUS probe 31 being inserted, along a guide wire 32,through a guiding catheter 33. Guiding catheter 33 is typically insertedthrough a sheath 34 and is connected to a Y connector 35. FIG. 2B showssensor 36 disposed between guiding catheter 33 and Y connector 35.Sensor 36 measures the longitudinal motion of a proximal portion of IVUSprobe 31 into (e.g., during insertion) and/or out of (e.g., duringpullback/withdrawal) guiding catheter 33. For some applications, thesensor is optical (e.g., laser-based), mechanical, electric, magnetic,or any combination thereof. For some applications, in response tomeasuring the longitudinal motion of the proximal portion of the IVUSprobe, the system estimates a distance by which the data-acquisitionportion of the IVUS probe has moved along a the lumen (typically, alongthe center line of the lumen), typically automatically and typicallyon-line. The center line is determined, typically automatically, inaccordance with techniques described in US 2010/0228076 to Blank, whichis incorporated herein by reference.

It is noted that, for some applications, sensor 36 is used for otherendoluminal applications in which a luminal roadmap is generated, andsubsequently the sensor is used for determining the current location ofan endoluminal tool along the roadmap. For some applications, thelocation of the endoluminal tool is determined while the endoluminaltool is not being imaged by extraluminal imaging. For some applications,the roadmap is generated and/or utilized in accordance with techniquesdescribed in US 2008/0221442 to Tolkowsky, which is incorporated hereinby reference. For some applications, the roadmap is generated and/orutilized in accordance with techniques described in US 2010/0160764 toSteinberg, which is incorporated herein by reference.

Reference is now made to FIG. 3, which is a flow chart, at least some ofthe steps of which are used in the course of co-use of endoluminal data(e.g., generated by an IVUS probe) and extraluminal imaging (e.g.,fluoroscopic imaging), in accordance with some applications of thecurrent invention. It is noted that although the steps described withreference to FIG. 3 are described with reference to IVUS imaging, thescope of the present invention includes applying these steps to anyother forms of endoluminal data-acquisition. It is noted that, for someapplications, some of the steps shown in FIG. 3 may be practiced withoutall of the steps shown in FIG. 3 necessarily being practiced incombination.

In phase 1, an IVUS probe is inserted to the site of an occlusion underfluoroscopic imaging, to acquire images of the endoluminal anatomy.

In phase 2, the fluoroscopic image stream is typically stabilized. Forsome applications, the image stream is stabilized with respect toradiopaque segments of the IVUS probe.

In phase 3, the IVUS probe is stopped at a location that is distal tothe designated luminal site (the designated site being the site of alesion, for example, as described hereinabove).

In phase 4, contrast agent is injected and an angiogram sequence isgenerated, under fluoro or cine.

In phase 5, an initial best angiogram frame is selected, typicallyautomatically and typically on-line. The initial best angiogram frame istypically selected based upon the following criteria: (a) the frame isacquired at a desired cardiac phase (typically end diastole) (b) in theimage frame, contrast agent highlights the vessel, and (c) radiopaqueelements (such as markers) at the distal section (i.e., in the vicinityof the imaging sensor) of the IVUS probe are visible in the image frame.

Reference is now made to FIG. 4, which shows an initial best angiogramof lumen 21, in accordance with some applications of the presentinvention. As shown in FIG. 4, radiopaque markers 22 of the IVUS probeare typically seen distally to lesion 23 in the initial best angiogram.

In phase 6, pullback of the IVUS probe, typically at a known and steadyrate of distance per second (such as by means of automated pullback),commences. The image slices generated by the IVUS probe along thepullback are recorded and stored in an image sequence. For someapplications, the pullback is performed manually.

For some applications, the image slices generated by the IVUS probealong the pullback are recorded and stored in an image sequence, andsimultaneously, a longitudinal position/movement sensor attached toapparatus through which the IVUS probe is inserted measures thelongitudinal location of a proximal portion of the IVUS probe relativeto the starting location of the proximal portion of the probe, e.g., asdescribed with reference to FIG. 2. The locations of the IVUS probe asdetermined by the sensor, when respective IVUS image slices wererecorded, are stored by the system.

For some applications, in the course of pullback, the lumen alsoexperiences cyclical motion (typically due to the cardiac cycle) thatcauses it to pulsate and move back and forth relatively to the IVUSprobe. For some applications, e.g., in the case of a lumen thatundergoes such back-and-forth cyclical motion, data acquired by the IVUSprobe is gated to the cyclical motion cycle of the lumen. Subsequently,IVUS images acquired in the course of the pullback at at-least-onespecific phase (for example, an end-diastolic phase) of the motion cycleof the lumen are co-registered with one or more fluoroscopic imagesacquired, and gated, at the corresponding at-least-one phase during thepullback, in order to facilitate the co-registration of the IVUS imagesto the fluoroscopic images. For some applications, co-registering IVUSimages with angiographic images that are gated to the same phase as thephase to which the IVUS images were gated, reduces distortions to theco-registration that may be introduced due to the cyclical motion of thelumen in the absence of using the aformentioned gating techniques.

For some applications, as described hereinbelow, in a subsequent phase(e.g., phase 13), in response to the user indicating location on theextraluminal image, the system retrieves and displays a correspondingendoluminal image frame, typically automatically and typically on-line.For some applications, the system displays the closest gated endoluminalimage frame corresponding to the location indicated by the user, eventhough there may be a non-gated image frame the location of which moreclosely corresponds to the location indicated by the user.Alternatively, the system displays the endoluminal image frame thelocation of which most closely corresponds to the location indicated bythe user, irrespective of the phase of the cardiac cycle at which theendoluminal image frame was acquired.

For some applications, there is a single, gated extraluminal angiogramimage to which all gated endoluminal data are co-registered. For someapplications, a three-dimensional model is generated from two (or more)two-dimensional gated angiograms, and the gated endoluminal data isco-registered with that three-dimensional model.

For some applications, the commencement of pullback is identified,typically automatically and typically on-line, by means of imageprocessing. For some applications, the image processing is performed byan image comparator which identifies a significant change (such as inthe color of image pixels or in the geometry of image features) in thesequentially-acquired endoluminal images, and interprets the change asindicating the commencement of pullback. For some applications, thecommencement of pullback is identified by means of a signal transmittedby the pullback unit and/or by the endoluminal data acquisition system.For some applications, the commencement of pullback is indicated bymeans of user input.

In phase 7, the pullback is stopped at a location that is proximal tothe designated lesion. For some applications, the termination ofpullback is identified, typically automatically and typically on-line,by means of image processing. For some applications, the imageprocessing is performed by an image comparator which identifies adiminishing change in the sequentially-acquired endoluminal images, andinterprets the diminishing change as indicating the termination ofpullback. For some applications, the termination of pullback isidentified by means of a signal transmitted by the pullback unit and/orby the endoluminal data acquisition system. For some applications, thetermination of pullback is indicated by means of user input.

In phase 8, contrast agent is injected and an angiogram sequence, underfluoro or cine, is generated.

In phase 9, a post-pullback best angiogram frame is selected, typicallyautomatically and typically on-line. The post-pullback best angiogramframe is typically selected based upon the following criteria: (a) theframe is acquired at a desired cardiac phase (typically end diastole)(b) in the image frame, contrast agent highlights the vessel, and (c)radiopaque elements (such as markers) at the distal section (i.e., inthe vicinity of the imaging sensor) of the IVUS probe are visible in theimage frame.

Reference is now made to FIG. 5, which shows a post-pullback bestangiogram of lumen 21, in accordance with some applications of thepresent invention. As shown in FIG. 5, radiopaque markers 22 of the IVUSprobe are typically seen proximally to lesion 23, in the post-pullbackbest angiogram.

In phase 10, the initial best angiogram and the post-pullback bestangiogram are co-registered to one another, typically automatically andtypically on-line, according to techniques described in US 2010/0222671to Cohen, which is incorporated herein by reference. A combined bestangiogram is generated by co-registering the initial and post-pullbackbest angiograms. Typically, in the combined best angiogram, the vesseland two sets of the IVUS probe's radiopaque elements (one of the setsbeing from the initial best angiogram and the second set being from thepost-pullback best angiogram) are visible.

Alternatively, the combined best angiogram is generated by adding themarkers that are visible in the initial best angiogram onto thepost-pullback best angiogram using the aforementioned registrationtechniques. Further alternatively, the combined best angiogram isgenerated by adding the markers that are visible in the post-pullbackbest angiogram onto the initial best angiogram using the aforementionedregistration techniques.

Reference is now made to FIG. 6, which shows a combined best angiogramof lumen 21, in accordance with some applications of the invention. Asshown in FIG. 6, distal radiopaque markers 22 of the IVUS probe areshown (e.g., overlaid) at their locations before and after the pullbackon the combined best angiogram of the lumen.

In phase 11, a center line typically is generated on the combined bestangiogram (for example, in accordance with the techniques described inUS 2010/0220917 to Steinberg, which is incorporated herein by reference)from the proximal to the distal marker locations along the vessel. Thesystem generates an index of the IVUS slices, based upon the estimatedlocation of the IVUS probe marker (from the distal-most marker locationthe proximal-most marker location) along the lumen (and typically alongthe center-line), at the time of acquisition of respective slices.

For some applications, the system interpolates between the distal-mostlocation of the IVUS marker along the lumen (e.g., along the center lineof the lumen) and the proximal-most location of the IVUS marker alongthe lumen (e.g., along the center line), in order to determine thelocation of the IVUS marker corresponding to intermediate IVUS slices.For some applications, in indexing the IVUS slices between theproximal-most and distal-most slices, it is assumed that pullback of theIVUS probe was performed at a linear rate, and that there is thereforean equal distance between any pair of adjacent IVUS slices, and anyother pair of adjacent IVUS slices (i.e., it is assumed that betweenacquiring respective successive pairs of slices, the probe traveledequal distances). For some applications, in indexing the IVUS slicesbetween the proximal-most and distal-most slices, the system accountsfor the IVUS probe acquiring IVUS images at varying frame rates.

In phase 12, the IVUS probe is retrieved.

In phase 13, while observing angiographic images of the luminal segmentcomprising the designated location, one or more locations along thatsection are indicated by a user input device. Typically, the userdesignates a location using the user input device, and the systemidentifies a location along the lumen (typically, along the luminalcenter line) as corresponding to the designated location, and retrievesthe previously-acquired IVUS images corresponding to the location, basedupon the indexing of the IVUS frames. The retrieved endoluminal imageframes previously recorded at the selected location are displayed.

Alternatively, by observing an angiogram frame side by side withendoluminal image frames of the luminal segment comprising thedesignated location, one or more locations along the section areindicated by a user input device with respect to endoluminal imagingdata. For some applications, the user indication is made upon theendoluminal image stack. For some applications, the user indication ismade by browsing through the endoluminal images. In response toreceiving the user indication, the location along the lumen (e.g., alongthe luminal center line) within the angiogram corresponding to thelocation indicated with respect to an endoluminal image or theendoluminal image stack is determined and indicated. For someapplications, the corresponding location on the angiogram is determinedbased upon the indexing of the IVUS slices, based upon the location ofthe IVUS probe marker along the lumen (e.g., along the luminalcenter-line), at the time of acquisition of respective slices, asdescribed hereinabove with reference to phase 11.

For some applications, the location corresponding to the locationindicated with respect to the endoluminal image frames or theendoluminal image stack is displayed on the combined best angiogram.Alternatively, (in cases in which a plurality of angiograms are acquiredduring pullback, as described hereinbelow) the location is displayed onthe angiogram that was acquired closest in time to the acquisition ofthe endoluminal image frame indicated by the user input device.

Reference is now made to FIG. 7, which is schematic illustration of ascreen on which an IVUS image 83 is displayed, in accordance with someapplications of the present invention. Typically, upon receiving anindication from the user of a location along the lumen (e.g., along theluminal center line 82, for example, by the user pointing cursor 81 to alocation on the screen, and the system determining a location along thecenter line corresponding to the location), IVUS image 83 which waspreviously acquired at that location is displayed. For someapplications, an IVUS stack comprising data from IVUS images that werepreviously acquired along a section of the lumen (e.g., along a sectionof center line 82) of which the user-indicated location is a middlepoint or one of the end points, is displayed. For some applications, anIVUS stack comprising data from IVUS images that were previouslyacquired between two user-indicated locations along the lumen (e.g.,along center line 82) is displayed.

Typically, a clinical diagnosis is facilitated by an operator viewingpreviously-acquired endoluminal images corresponding to the one or morelocations selected on extraluminal images of the luminal segment, or bythe operator viewing indications of locations on an extraluminal imagethat correspond to one or more locations selected on endoluminal images,as described with reference to phase 13. Alternatively, a clinicaldiagnosis is made by the operator reviewing the extraluminal imagesand/or the endoluminal data (and/or reviewing other data), withoutperforming phase 13. Typically, a therapeutic process, such as the onedescribed in phase 14 and beyond, is performed based upon the clinicaldiagnosis made by the operator.

In phase 14, a catheter with a balloon and/or stent is inserted to thearea of the designated site, under fluoroscopic imaging. Typically, thefluoroscopic image stream is stabilized with respect to radiopaquemarkers on the catheter via which the balloon and/or the stent isinserted.

In phase 15, upon reaching a desired location within the blood vessel(such as the vicinity of the designated site), contrast agent isinjected and an angiogram sequence is generated under fluoro or cine.

In phase 16, a current best angiogram frame is selected, typicallyautomatically and typically on-line. The current best angiogram frame istypically selected based upon the following criteria: (a) the frame isacquired at a desired cardiac phase (typically end diastole) (b) in theimage frame, contrast agent highlights the vessel, and (c) radiopaqueelements (such as markers) at the distal section (i.e., in the vicinityof the imaging sensor) of the balloon/stent catheter are visible in theimage frame.

In phase 17, the combined best angiogram and the current best angiogramare co-registered to one another, typically automatically and typicallyon-line, according to techniques described in 2010/0222671 to Cohen,which is incorporated herein by reference. A multi-combined bestangiogram is generated by co-registering the combined and current bestangiograms. Typically, in the multi-combined best angiogram, the vesseland the two sets of the IVUS probe's radiopaque elements (one of thesets being from the initial best angiogram and the second set being fromthe post-pullback best angiogram) and the radiopaque markers of theballoon/stent catheter are visible.

In phase 18, a user and/or the system selects a location of interestalong the lumen in the multi-combined best angiogram of the lumen. Forexample, a user or the system may select a location of a point ofinterest along the balloon/stent (such as the location of one of theballoon/stent markers, or anywhere in between the markers).

In phase 19, based upon the indexing described hereinabove, withreference to phase 11, the IVUS image previously recorded at theselected location (which is typically based upon the current location ofthe balloon/stent catheter, as described in the previous step (stepxvi)) is identified. The corresponding IVUS image is retrieved anddisplayed, typically automatically and typically on-line, together withthe fluoroscopic images. For some applications, the IVUS images aredisplayed in a separate window (but on the same screen as thefluoroscopic images). For some applications, the IVUS images aredisplayed on a separate screen. For some applications, the IVUS imagesthat are displayed are two-dimensional (also known as “slices”). Forsome applications, a stack comprising multiple IVUS slices (such asthose corresponding to the longitudinal section between the currentlocations of the proximal and distal markers of the balloon/stent, and,optionally, beyond the aforementioned current marker locations, in eachdirection) is displayed.

For some applications, a three-dimensional “tunnel-like” reconstructionof the IVUS images of the vessel (or a section thereof, such as thosecorresponding to the longitudinal section between the current locationsof the proximal and distal markers of the balloon/stent) is generatedand displayed. For some applications, the IVUS images are overlaid onthe fluoroscopic images. For some applications, the IVUS images arefused with the fluoroscopic images. For some applications, a combinationof the aforementioned display techniques is applied. For someapplications, an indication of the motion range of the balloon/stentrelative to the lumen, resulting from the cardiac cycle, is displayed inconjunction with any of the aforementioned displays of the IVUS images.For some applications, such an indication is generated and/or displayedin accordance with embodiments of US 2010/0222671 to Cohen, which isincorporated herein by reference.

As an alternative or in addition to phases 18 and 19, by observing anangiogram frame (e.g., the multi-combined best angiogram) side by sidewith endoluminal image frames of the luminal segment comprising thedesignated location, one or more locations along the section areindicated by a user input device with respect to endoluminal imagingdata. For some applications, the user indication is made upon theendoluminal image stack. For some applications, the user indication ismade by browsing through the endoluminal images. In response toreceiving the user indication, the location along the lumen (e.g., alongthe luminal center line) within the angiogram frame (e.g., the combinedbest angiogram) corresponding to the location indicated with respect tothe endoluminal image or the endoluminal image stack is determined andindicated.

In phase 20, as a result of displaying the IVUS image or an imagederived from the IVUS image (e.g., a fused image), the balloon and/orstent may be positioned and deployed based upon an on-line combinationof real-time fluoroscopic images and of IVUS images recorded earlier(for example, more than a minute earlier).

Although phases 14-20 have been described with respect to inserting aballoon and a stent into the lumen, the scope of the present inventionincludes performing steps 14-20 in conjunction with a differenttherapeutic device being inserted into the lumen, mutatis mutandis. Forexample, a guide wire may be inserted into the lumen in order topenetrate an occlusion (e.g., a total occlusion) of the lumen. Aradiopaque marker that is visible in extraluminal images (e.g.,fluoroscopic and/or angiographic images) is disposed at the distal endof the guidewire. By applying phases 14-20 in conjunction with theinsertion of the guidewire through the occlusion, the system facilitatesthe retrieval and display of endoluminal images (e.g., OCT and/or IVUSimages) of the lumen that correspond to the current location of theradiopaque marker of the guidewire. For some applications, the systemfacilitates the display of the current location of the radiopaque markerof the guidewire with respect to a previously-acquired endoluminal imagestack of the lumen.

Typically, in the case of a total occlusion of the lumen, it is notpossible to acquire endoluminal images of the occluded segment of thelumen, since the occluded segment is closed. For some applications, insuch cases, a forward-looking endoluminal imagining probe is used toacquire endoluminal images of segments of the lumen that are distal tothe probe, while the probe is at respective locations within the lumen.Subsequently, when a guidewire is inserted through the lumen, in orderto penetrate the occlusion, an endoluminal image of a segment of thelumen that is distal to the guidewire corresponding to the currentlocation of the guidewire is shown, using the co-registration techniquesdescribed herein. Alternatively, when the guidewire is inserted throughthe lumen the current location of the tip of the guidewire is displayedwith respect to an endoluminal image stack of the lumen, the stack beingbased upon the previously-acquired endoluminal images.

As described hereinabove with reference to FIG. 3, for someapplications, initial and post-pullback angiograms are generated inorder to determine the locations of the IVUS markers with respect to thelumen before and after pullback. Intermediate marker locationscorresponding to intermediate endoluminal images are indexed byinterpolating between distal and proximal marker locations that aredetermined based upon, respectively, the initial and post-pullbackangiograms. Typically, in indexing the IVUS slices between theproximal-most and distal-most slices, it is assumed that pullback of theIVUS probe was performed at a linear rate and that the frame rate of theIVUS probe was constant. It is therefore assumed that there is an equaldistance between any pair of adjacent IVUS slices, and any other pair ofadjacent IVUS slices.

Alternatively, as described hereinabove, the system estimates the speedof the pullback of the imaging head of the IVUS probe by measuring thespeed of the pullback of a proximal portion of the probe, using asensor, e.g., as described with reference to FIG. 2. Thus, the systemmay determine an initial location of the IVUS markers by acquiring aninitial angiogram, and may determine subsequent locations of the IVUSmarkers based upon the estimated speed of the pullback of the IVUS probeand the time that has elapsed between the commencement of pullback andthe estimated speed of the pullback.

Further alternatively, as described hereinbelow with reference to FIG.8, pullback of the endoluminal imaging probe is performed while thelumen is continuously flushed with contrast agent. This is applicable,for example, in the case of an endoluminal OCT probe, as describedhereinbelow. For example, in such cases, the lumen may be continuouslyflushed with contrast agent for a time period of at least two seconds,and/or for at least 50% (e.g., at least 80%) of the duration of a timeperiod over which the imaging probes acquires the endoluminal imagesduring pullback. In such cases, the entire pullback procedure (or anentire portion thereof) may be performed under angiographic imaging. Theendoluminal probe marker locations corresponding to given endoluminalimages are determined by identifying marker locations in theangiographic images (e.g., via image processing that is typicallyperformed automatically and on-line), co-registering the angiographicimages into a combined best angiogram, and indexing the identifiedmarker locations with respect to the endoluminal images, as describedhereinbelow. Intermediate marker locations that do not appear in thecombined best angiogram are estimated, as described hereinbelow, withreference to FIG. 8. The aforementioned technique may be typically usedto determine marker locations even in cases in which the pullback of theendoluminal imaging probe is not performed at a constant speed, sincemarker locations that are known are typically relatively close to oneanother.

Still further alternatively, endoluminal probe marker locationscorresponding to respective endoluminal images are determined, byacquiring fluoroscopic images of the probe within the lumen during thepullback (the fluoroscopic images typically being acquired withoutrequiring the injection of contrast materials). The endoluminal probemarker locations corresponding to given endoluminal images aredetermined by identifying marker locations in the fluoroscopic images(e.g., via image processing) and indexing the identified markerlocations with respect to the endoluminal images. Typically, theradiopaque markers of the probe are identified, typically automaticallyand typically on-line, and their locations are determined, typicallyautomatically and typically on-line, according to their distances alonga guide wire along which the probe is inserted. For some applications,the distances are measured relative to the distal tip of a guidingcatheter through which the guide wire was previously inserted.Alternatively, the marker locations are measured relative to otherportions of the apparatus that are visible in the fluoroscopic imagesand that are substantially stationary with respect to the lumen duringpullback of the probe, as described hereinabove with reference to phase5 of the flowchart shown in FIG. 1. The aforementioned technique may beused to determine marker locations even in cases in which the pullbackof the endoluminal imaging probe is not performed at a constant speed.

For some applications, the determination of marker locations isgenerally as described with reference to FIG. 3. That is, initial andpost-pullback angiograms are generated in order to determine thelocations of the IVUS markers with respect to the lumen before and afterpullback, and intermediate marker locations corresponding tointermediate endoluminal images are indexed by interpolating betweenknown marker locations. However, as described hereinabove, in indexingthe IVUS slices between the slices that were acquired at known markerlocations, it is typically assumed that pullback of the IVUS probe wasperformed at a linear rate, and that there is therefore an equaldistance between any pair of adjacent IVUS slices, and any other pair ofadjacent IVUS slices. For some applications, in order to overcome errorsin the estimated marker locations due to non-linear pullback of theprobe (and/or for a different reason), additional intermediate markerlocations are identified. Marker locations between the identified markerlocations are indexed by interpolating between the two closestidentified marker locations, and not just by interpolating between thedistal-most and proximal-most marker locations. Typically, such atechnique reduces errors in estimating the intermediate marker locationsdue to a non-linear pullback rate of the endoluminal imaging probe,relative to a technique in which only the distal-most and proximal-mostmarker locations are identified.

For some applications, the additional marker locations are determined byacquiring additional angiograms in between the acquisition of theinitial angiogram and the post-pullback angiogram. For each of theseangiograms, the endoluminal imaging probe marker is identified, and thebest frame is selected, typically in accordance with the techniquesdescribed herein. The marker location is typically co-registered withthe combined best angiogram, in accordance with the techniques describedherein. Based on the marker locations that are derived from theintermediate angiograms, a plurality of known marker locations arethereby determined with respect to the combined best angiogram. Thesystem indexes IVUS slices at any section along the lumen (e.g., alongthe luminal center line) within the combined best angiogram byinterpolating the marker locations corresponding to respective IVUSslices with reference to the two closest known IVUS marker locations tothat section.

Alternatively, intermediate marker locations are determined byidentifying a feature in an endoluminal image that is also identifiablein the combined best angiogram (or in an angiogram that is co-registeredto the combined best angiogram). In response thereto, the location ofthe endoluminal probe marker at the acquisition of the endoluminal imagemay be determined with respect to the combined best angiogram. For someapplications, the feature is a bifurcation, a curve or some other uniqueshape, a partial or total occlusion, a native valve, an aneurism, aseptal defect, or a malformation. For some applications, the feature isa previously-deployed device visible in the extraluminal imaging. Forsome applications, the previously-deployed device is a stent, or agraft, or a replacement valve.

It is noted that in applying any of the techniques described hereinabovefor associating endoluminal images with respective locations along thelumen, the system typically accounts for a known offset between thelocation of the moving, visible portion of the endoluminal imaging probe(e.g., a radiopaque marker), and the location of the image-acquiringportion of the probe (e.g., the ultrasound transducer, in the case of anIVUS probe).

It is noted that some of the techniques described hereinabove forassociating endoluminal images with respective locations along the lumenare described with reference to an endoluminal imaging probe thatacquires endoluminal images during pullback of the probe. The scope ofthe present invention includes applying any of the techniques describedhereinabove for associating endoluminal images with respective locationsalong the lumen to an endoluminal imaging probe that acquiresendoluminal images during insertion and advancement of the probe throughthe lumen (e.g., when images are acquired from an endobronchial airway),mutatis mutandis.

In general, when applying the techniques described herein, in indexingendoluminal image frames at any point along the lumen (e.g., along theluminal center line) with reference to the two closest identified markerlocations to the point, it is typically assumed that pullback of theendoluminal probe and the acquisition of images by the endoluminal probewere performed at linear rates, and that there is therefore an equaldistance between any pair of adjacent endoluminal images, and any otherpair of adjacent endoluminal images that were acquired between the twoclosest identified locations of the endoluminal probe. For someapplications, in indexing the endoluminal images at any point along thelumen (e.g., along the luminal center line) with reference to the twoclosest identified marker locations to the point, the system accountsfor the probe acquiring endoluminal images at varying frame rates,and/or for pullback being performed at a non-linear rate (the rate ofpullback in such cases, typically being estimated, based uponmeasurements of a sensor, as described with reference to FIG. 2).

For some applications, techniques described herein (e.g., techniquesdescribed with reference to FIGS. 1-11) are performed by a system thatincludes at least one processor. The processor is typically for use withan endoluminal data-acquisition device configured to acquire a pluralityof endoluminal data points of a lumen of a body of a subject atrespective locations inside the lumen, while the endoluminaldata-acquisition device is moved through the lumen, the endoluminaldata-acquisition device having a radiopaque marker coupled thereto. Theprocessor is typically for use with an angiographic imaging deviceconfigured to acquire respective angiographic image of the lumen, attimes associated with acquisitions of respective endoluminal data pointby the endoluminal data-acquisition device. For some applications, theprocessor includes location-association functionality configured todetermine first and second locations of the radiopaque markerrespectively within first and second angiographic images of the lumen.For some applications, the processor includes image-co-registrationfunctionality configured to generate a combined angiographic image ofthe lumen that includes representations of the first and second markerlocations thereon, by co-registering the first and second angiographicimages. For some applications, the processor includeslocation-association functionality configured to determine that at leastone location on the combined angiographic image that is intermediate tothe first and second locations of the radiopaque marker corresponds toan endoluminal data point acquired between the acquisitions of first andsecond data points corresponding to the first and second locations ofthe marker, by interpolating between the first and second locations ofthe radiopaque marker on the combined angiographic image. Typically, theprocessor includes display-driving functionality configured to drive thedisplay to display an output, in response to determining that theintermediate location corresponds to the endoluminal data point acquiredbetween the acquisitions of the first and second data points.

For some applications, the image-co-registration functionality isconfigured to generate the combined angiographic image of the lumen thatincludes representations of the first and second marker locationsthereon, by co-registering the first and second angiographic images toone another, by designating one of the angiographic images as a baselineimage, a shape of the lumen in the baseline image being designated as abaseline shape of the lumen. The image-co-registration functionalitytypically determines whether a shape of the lumen in the angiographicimage that is not the baseline image is the same as the baseline shapeof the lumen, and in response to determining that the shape of the lumenin the angiographic image that is not the baseline image is not the sameas the baseline shape of the lumen designates the image that is not thebaseline image as a non-baseline image. The image-co-registrationfunctionality typically deforms the shape of the lumen in thenon-baseline image, such that the shape of the lumen becomes moresimilar to the baseline shape of the portion than when the lumen in thenon-baseline image is not deformed, and based upon the deformation ofthe non-baseline image, determines a location upon the baseline image atwhich the marker from within the non-baseline image should be located.The image-co-registration functionality typically generates anindication of the marker from within the non-baseline image at thedetermined location on the baseline image. For some applications, theimage-co-registration functionality is configured to generate thecombined angiographic image of the lumen using similar techniques tothose described in US Patent Application 2010/0172556 to Cohen et al.,which is incorporated herein by reference.

For some applications, techniques described herein (e.g., techniquesdescribed with reference to FIG. 3) are performed by a system thatincludes at least one processor. The processor is typically for use with(a) an endoluminal data-acquisition device configured to acquire aplurality of endoluminal data points of a lumen of a body of a subjectat respective locations inside the lumen, while the endoluminaldata-acquisition device is being moved through the lumen, theendoluminal data-acquisition device having a radiopaque marker coupledthereto, (b) contrast agent configured to be continuously injected intothe lumen, during the movement of the endoluminal data-acquisitiondevice, and (c) an angiographic imaging device configured to acquire aplurality of angiographic images of the endoluminal data-acquisitiondevice inside the lumen, during the movement of the endoluminaldata-acquisition device. The processor typically includes (a)location-association functionality configured to determine thatendoluminal data points correspond to respective locations within thelumen, by determining locations of the radiopaque marker within theangiographic images of the lumen, by performing image processing on theangiographic images, the locations of the radiopaque marker within theangiographic images of the lumen corresponding to respective endoluminaldata points, and (b) display-driving functionality configured to drivethe display to display an output, in response to determining that theendoluminal data points correspond to respective locations within thelumen.

For some applications, phases 1 through 13 (or any applicable subset ofthose phases) of FIG. 3, are repeated subsequent to the deployment ofthe therapeutic device, such as in the course of performing a clinicalevaluation of the outcome of the deployment of that device. For example,phases 1-13 may be repeated so as to facilitate the co-display ofendoluminal images of the lumen, post-deployment of the device, with oneor more extraluminal images of the lumen.

For some applications, pullback of the endoluminal data-acquisitiondevice is performed in the course of a continuous injection of contrastagent performed under fluoroscopic imaging. For example, the endoluminaldata-acquisition device may be an OCT probe, the image acquisition ofwhich typically requires concurrent flushing of the lumen, in order toremove blood from the lumen, the blood interfering with the OCT imaging.Furthermore, contrast agent highlights the lumen and facilitatesangiographic imaging of the lumen. Still furthermore, for someapplications, the presence of contrast agent in the lumen facilitatesacquisition of OCT data. Therefore, typically, during endoluminalimaging with an OCT probe, contrast agent is continuously injected intothe lumen. In addition, the pullback of the OCT probe is typicallyperformed rapidly relative to the pullback of an IVUS probe, and theframe acquisition rate of the OCT probe is typically greater than thatof an IVUS probe.

For endoluminal imaging techniques such as OCT techniques, in whichpullback of the imaging probe is performed under constant angiographicimaging, phases 4 to 10 of the technique described with reference to theflowchart shown in FIG. 3 may be substituted or combined with the phasesdescribed below. The steps described below are typically performed inconjunction with at least some of the other phases described withreference to the flowchart shown in FIG. 3, mutatis mutandis. Althoughthe steps below are described with reference to endoluminal imaging withan OCT probe, the scope of the present invention includes performingthese steps when using a different endoluminal imaging probe (such as anIVUS probe), the pullback of which is performed under constantangiographic imaging.

Pullback of the OCT probe commences (for example, by means of manualpullback, or at a known and steady rate of distance per second, such asby means of automated pullback), in conjunction with contrast agentinjection performed under fluoroscopic imaging. The image slicesgenerated by the OCT along the pullback are recorded and stored,synchronized (such as by time or by frame number) with the correspondingstored angiographic images. In addition, the locations of the OCTmarkers corresponding to respective, at least some OCT image slices arestored with reference to the corresponding stored angiographic image.For some applications, the marker locations are determined byidentifying the markers in the angiographic images by (typicallyautomatically) performing image processing on the angiographic images.

The total number of OCT images and fluoroscopic images acquired duringthe pullback may differ (due to different image acquisition framerates). For example, the fluoroscopy frame rate may be 25 frames persecond, whereas the OCT frame rate may be 100 frames per second, inwhich case OCT frames 1 through 4 are indexed to fluoroscopy frame 1,OCT frames 5 through 8 are indexed to fluoroscopy frame 2, etc.

From along the contrast injection in the course of the pullback, thesystem selects an angiogram frame, typically according to criteriadescribed hereinabove, and depicts upon that frame the locations of theradiopaque marker(s) of the OCT probe, in whole or in part, by means ofimage processing, during pullback. The selected angiogram is denoted asthe combined best angiogram. For some applications, and typicallypursuant to the selection of the combined best angiogram, non-rigidtransformation of one or more angiogram frames from the pullbacksequence to the combined best angiogram is performed, typicallyautomatically and typically on-line. The non-rigid transformation istypically followed by the depiction of the locations of the radiopaquemarker(s) of the OCT probe on the resulting combined best angiogram,typically automatically and typically on-line. In depicting the markerlocations on the resulting combined best angiogram, the non-rigidtransformation of angiographic image frames associated with respectivemarker locations is accounted for. For some applications, such non-rigidtransformation and marker depiction are performed according totechniques described in 2010/0222671 to Cohen, which is incorporatedherein by reference.

Reference is now made to FIG. 8 which shows a combined best angiogram,the combined best angiogram having been created in the course of asequence for use in conjunction with an OCT endoluminal imaging probe,as described hereinabove, in accordance with some applications of thepresent invention. As shown, known OCT probe marker locations are shownon the combined best angiogram. Typically, even in cases in whichpullback is performed under continuous angiographic imaging, not all ofthe marker locations are known, since the frame rate of the OCT probe istypically greater than the frame rate of the x-ray imager. Theendoluminal probe marker locations corresponding to given endoluminalimages that are not identifiable in the angiographic image aredetermined by indexing the marker locations with respect to theendoluminal images.

For example, in order to determine which is the endoluminal imagecorresponding to the point along the lumen in the combined bestangiogram indicated by the double arrow in FIG. 8, it is determined thatthe OCT probe marker is a given distance between the marker locationsthat are identifiable in a given pair of angiograms. Based upon thetimes of the acquisitions of the given pair of angiograms, the rate atwhich the angiograms were acquired, and the rate at which the OCT frameswere acquired, the OCT frame corresponding to that point may bedetermined.

For example, if the frame rate of the angiograms is 25 per second, thepair of angiograms were acquired, respectively at 0.7 seconds and 1.8seconds from a given starting time, and the indicated location is onethird of the distance between the marker locations known from the pairof angiograms, then it is determined that the corresponding OCT imagewas acquired at 1.06 seconds after the starting time, by performing thefollowing calculation:

(0.33*(1.8−0.7)+0.7)=1.06

Thus, if the frame rate of the OCT probe is 100 frames per second, thecorresponding OCT frame is frame 106.

Reference is now made to FIG. 9, which shows the co-display ofpreviously-acquired endoluminal image frames (e.g., frame 91), theendoluminal locations of the endoluminal imaging probe at the time ofthe acquisition of respective image frames being indicated on anextraluminal image of the lumen, the locations having been automaticallyindentified during pullback of the endoluminal imaging probe, inaccordance with some applications of the present invention. For someapplications, previously-acquired endoluminal OCT images frames (orother forms of endoluminal image frames) are connected by lines, to thecorresponding endoluminal locations (such as location 92), of the OCTimaging probe at the times that the OCT image frames were acquired. Forsome applications, the range of the pullback is indicated with respectto an OCT image stack 94. For example, a line 93 is generated on the OCTimage stack indicating where the pullback ended. For some applications,some endoluminal locations, such as location 92, are indicated as beingassociated with a corresponding location on the endoluminal image stack.For example, a line that is similar to line 93 may be generated on OCTimage stack 94 to indicate the location on the image stack thatcorresponds to location 92.

For some applications, data acquired by a first endoluminal modality(e.g., IVUS) are co-registered with the fluoroscopic image stream, inaccordance with the applications described hereinabove. Subsequently,data acquired by a second endoluminal modality (e.g., OCT) areco-registered with the fluoroscopic image stream, in accordance with theapplications described hereinabove. Consequently, due to both data setsbeing co-registered with the fluoroscopic image stream, the two datasets are co-registered to one another. For some applications, the twoendoluminal data sets are displayed as overlaid or otherwise merged withone another.

For some applications, generally similar steps to those described withreference to FIG. 3 are performed, except for the following differences.In phase 14, instead of a therapeutic endoluminal device (e.g., atreatment catheter) being inserted into the lumen, a second endoluminaldata-acquisition device is inserted into the lumen. Typically, the firstand second endoluminal data-acquisition devices acquire endoluminalimages using respective imaging modalities. For example, in phase 1, anIVUS probe may be inserted into the lumen, and in phase 14 an OCT probemay be inserted into the lumen, or vice versa.

The current location of the second endoluminal data-acquisition deviceis determined, for example, using any of the techniques described herein(such as, by performing image processing on extraluminal images of thesecond endoluminal data-acquisition device inside the lumen).Endoluminal images which were previously acquired using the firstdata-acquisition device at the current location of the secondendoluminal data-acquisition device are retrieved and displayed,typically on-line and typically automatically.

Typically, the endoluminal images which were acquired using the firstdata-acquisition device at the current location of the secondendoluminal data-acquisition device are displayed together withendoluminal images that are being acquired in real time by the secondendoluminal data-acquisition device, while the second endoluminaldata-acquisition device is at the current location. For someapplications, endoluminal images that are acquired in real time by thesecond endoluminal data-acquisition device, while the second endoluminaldata-acquisition device is at the current location, are displayedtogether with an indication of the current location of the secondendoluminal data-acquisition device with respect to an endoluminal imagestack generated using endoluminal images that were previously acquiredby the first endoluminal data-acquisition device. For some applications,using the above-described technique, data acquired by first and secondendoluminal data acquisition devices are registered with respect to oneanother, and the co-registered data are displayed subsequent totermination of the acquisition of endoluminal images by both the firstand the second endoluminal data-acquisition devices. For someapplications, endoluminal images corresponding to the current locationof the second endoluminal data-acquisition device that were acquired bythe first endoluminal data acquisition device and/or by the secondendoluminal data acquisition device are co-displayed with an indicationof the current location of the second endoluminal data-acquisitiondevice on an extraluminal image of the lumen, using the techniquesdescribed herein.

Reference is now made to FIG. 10, which shows the co-use ofpreviously-acquired IVUS images and a current, stabilized, extraluminalfluoroscopic image stream, or with an angiogram image from the nativefluoroscopic image stream, in accordance with some applications of thepresent invention. The native fluoroscopic image stream is displayed inleft side window 101. A region of interest (ROI) 102 is marked, with adotted white line, within left side window 101. A stabilized imagestream, generally based upon ROI 102, is displayed in right side window103. Vessel 104 is highlighted, by means of contrast agent. Radiopaquemarkers 105 and 106 are mounted respectively at the proximal and distalends of a balloon carrying a stent. The balloon is being insertedthrough vessel 104. The balloon, as shown, is being positioned inpreparation for the deployment of the stent at partial occlusion 107which is at a narrower segment of vessel 104. An IVUS slice, acquiredprior to placement of the balloon with markers 105 and 106 into vessel104, corresponding to the current location of distal marker 106, isretrieved and displayed, typically in real time and typicallyautomatically, at the upper right corner of right side window 103. FIG.10 shows an illustrative IVUS slice 108 displayed in the upper rightcorner of right side window 103. In accordance with the applicationsdescribed hereinabove, the IVUS slice that is displayed is a slice thatwas acquired by an IVUS probe previously, while the probe was insertedinto the vessel under extraluminal fluoroscopy. IVUS slice 108 depicts ahealthy vessel location. The display of slice 108 concurrently withpositioning of the balloon, in preparation for stent deployment, assistsin confirming that the distal end of the stent (corresponding to distalmarker 106) is properly positioned at a “healthy shoulder” of occlusion107 (i.e., the point along the arterial lumen at which the occlusion isno longer significant and/or the disease is no longer prevalent), as istypically desired. For some applications, the display of thecorresponding IVUS slices is made relative to marker locations in asingle angiogram frame. For some applications, the display of thecorresponding IVUS slices is made relative to a three-dimensional modelthat was generated from two (or more) two-dimensional gated angiograms.

The cumulative effect of showing the extraluminal image stream and IVUSslice 108 is as if the stent is being positioned concurrently under bothextraluminal fluoroscopic imaging and endoluminal IVUS imaging. Inpractice, such concurrent imaging is typically not possible becausevessel 104 is too narrow to accommodate both the IVUS catheter and thestent catheter, and also because even if there were sufficient space,then the two catheters may interfere with one another.

Reference is now made to FIG. 11, which shows the co-use ofpreviously-acquired IVUS images and a current, stabilized, extraluminalfluoroscopic image stream, in accordance with some applications of thepresent invention. Stack 111 comprises previously-acquired IVUS slicespreviously acquired at locations corresponding to the current locationsof balloon markers 112 and 113. For some applications, the display ofthe corresponding IVUS stack is made relative to marker locations in astatic angiogram frame.

Reference is now made to FIG. 12, which is a graph showing the locationalong a lumen (e.g., along the center line of the lumen) of an imaginghead of an endoluminal imaging probe, versus the frame numbers of theendoluminal image frames acquired by the probe, during pullback of theprobe. Typically, even during automated pullback of the probe, therelative speed at which the imaging head of the probe moves with respectto the lumen, and, in some cases, the direction in which the imaginghead moves with respect to the lumen, varies over the course of thecardiac cycle, due to pulsation of the lumen. As shown on portion 115 ofthe graph (which typically corresponds to a systolic phase of thecardiac cycle, or a portion thereof), in some cases, the imaging head ofan endoluminal imaging probe moves forward (i.e., distally) with respectto the lumen during certain phases of the cardiac cycle, even duringpullback (pullback generally being in a distal to proximal direction).

Still further typically, as a result of the imaging head moving forwardwith respect to the lumen, in some cases, two or more endoluminal imageframes are acquired at a single location along the lumen. For example,as shown in FIG. 12, frames x, y, and z are acquired at a singlelocation along the lumen. Frame x is acquired pre-systole, while theprobe is moving in a distal to proximal direction with respect to thelumen, frame y is acquired during systole, while the probe is moving ina proximal to distal direction with respect to the lumen, and frame z isacquired post-systole, while the probe is moving back past the samelocation in a distal to proximal direction with respect to the lumen.

For some applications, manual pullback of the endoluminal imaging probeis performed by an operator. In some cases, during manual pullback, theoperator pushes the probe forward at times in order to view a givenregion for a second time. As a result, the imaging probe typicallyacquires a plurality of endoluminal images of given locations within theregion. For example, a first image may be acquired during the initialpullback past the location in the distal to proximal direction, a secondimage may be acquired when the probe is pushed forward by the operatorin the proximal to distal direction, and a third image may be acquiredwhen the probe is, subsequently, pulled back past the location in thedistal to proximal direction for a second time.

For some applications, forward motion of the endoluminal imaging probethat is (a) due to pulsation of the lumen, and/or (b) due to an operatorof the probe pushing the probe forward, is accounted for in order tofacilitate co-registration of the endoluminal images to an extraluminalimage. Typically, in order to facilitate co-registration, the systemidentifies redundant image frames (i.e., image frames that are notrequired because they are acquired at a location at which one or moreadditional image frames are acquired), and rejects at least some of theredundant image frames from being used for the co-registration, asdescribed in further detail hereinbelow.

For some applications, forward motion of the imaging probe is detectedby acquiring images of the imaging probe within the lumen, andperforming image processing on the angiographic images in order todetermine locations of the endoluminal image probe marker with respectto the lumen at the time of the acquisition of respective endoluminalimage frames, e.g., in accordance with the techniques describedhereinabove.

For some applications, angiographic images of the imaging probe withinthe lumen are acquired in the presence of contrast agent (which makesthe lumen visible in the angiographic images), and the angiographicimages are image processed in order to determine locations of theendoluminal image probe marker with respect to the lumen at the time ofthe acquisition of respective endoluminal image frames. Typically, usingimage processing of angiographic images of the probe within the lumencan be used to identify forward motion of the imaging probe that is (a)due to pulsation of the lumen, and (b) due to an operator of the probepushing the probe forward. This is because, in the angiographic images,the system typically identifies a visible moving portion of theendoluminal imaging probe (e.g., a radiopaque marker on the imaginghead). Using image processing, the system tracks the motion of thevisible, moving portion of the endoluminal probe with respect to thelumen. Thus, motion of the visible, moving portion of the imaging probewith respect to the lumen is identifiable in the angiographic images,irrespective of the cause of the motion.

For some applications, fluoroscopic images of the imaging probe withinthe lumen are acquired in the absence of contrast agent, and thefluoroscopic images are image processed in order to determine locationsof the endoluminal image probe marker with respect to the lumen at thetime of the acquisition of respective endoluminal image frames. For someapplications, as described hereinabove, the location of a moving,visible portion of the endoluminal imaging probe (e.g., a radiopaquemarker on the imaging head of the endoluminal imaging probe) isdetermined according to its distance along a guide wire along which theimaging probe is inserted, the distance typically being measuredrelative to the distal tip of a guiding catheter through which theguidewire and the imaging probe were previously inserted. For someapplications, the endoluminal imaging probe includes a portion thatsubstantially does not move with respect to the lumen during pullback,such as an insertion sheath. The location of moving, visible portion ofthe imaging probe is determined, via image processing, with reference tothe portion of the device that substantially does not move with respectto the lumen during pullback. Typically, using image processing offluoroscopic images of the probe within the lumen can be used toidentify forward motion of the imaging probe that is due to an operatorof the probe pushing the probe forward. However, image processing offluoroscopic images of the probe inside the lumen typically cannot beused to identify forward motion of the imaging probe that is due topulsation of the artery, since all of the components of the probe(including the guidewire and the insertion sheath, for example) movewith respect to the lumen due to pulsation of the lumen.

For some applications, forward motion of the endoluminal probe that iscaused by an operator pushing the probe forward is determined using alongitudinal position/movement sensor coupled to apparatus through whichthe endoluminal probe is inserted, e.g., as described hereinabove withreference to FIG. 2.

In response to determining that two or more endoluminal image framescorrespond to the same location along the lumen due to forward motion ofthe probe with respect to the lumen, at least one of the image frames isnot used for the co-display of the endoluminal image frames with anextraluminal image of the lumen. Typically, only the first endoluminalimage frame that was acquired at the location is used for the co-displayof the endoluminal image frames with an extraluminal image of the lumen.For some applications, it is determined which at least one of the two ormore endoluminal image frames that correspond to the same location alongthe lumen was acquired during forward motion of the probe, and thisframe is rejected from being used in the co-display. Alternatively oradditionally, another at least one of the two or more endoluminal imageframes that correspond to the same location along the lumen is rejectedfrom being used in the co-display.

For some applications, during pullback of the endoluminal imagingdevice, the subject's ECG signal is detected. Respective endoluminalimages are identified as corresponding to the period in the subject'scardiac cycle at the time when the image was acquired, based upon thedetected ECG signal (e.g., by indexing the image frames with respect tothe subject's ECG signal). For some applications, based upon theidentified correspondence, the system determines which of theendoluminal images were acquired in a given period of the subject'scardiac cycle, such as at least a portion of systole, and these imageframes are not used for the co-display of the endoluminal image frameswith an extraluminal image of the lumen. For example, framescorresponding to at least a portion of the subject's ECG signal betweenthe S and T waves may be rejected from being used in the co-display.Typically, associating endoluminal image frames with phases of thesubject's cardiac cycle (e.g., by indexing with respect to the subject'sECG signal) can be used to account for forward motion of the endoluminalimaging probe that is caused by motion of the probe with respect to thelumen due to pulsation of the lumen that is due to the subject's cardiaccycle.

For some applications, techniques described herein are used to accountfor the forward motion of the endoluminal imaging probe in order tofacilitate the generation of an endoluminal image stack, the forwardmotion of the imaging probe typically being (a) due to pulsation of thelumen, and/or (b) due to an operator of the probe pushing the probeforward. Typically, in order to facilitate generation of an endoluminalimage stack, the system identifies redundant image frames (i.e., imageframes that are not required because they are acquired at a location atwhich one or more additional image frames are acquired), and rejects atleast some of the redundant image frames from being used in theendoluminal image stack, as described in further detail hereinbelow. Forsome applications, in response to determining that some of the imageframes were acquired during forward motion of the imaging probe, thesystem places the image frames in order within the image stack, and/orre-orders frames in an image stack that has already been generated, suchthat the frames within the stack are placed in the correct order. Forsome applications, the system indicates image frames within an imagestack that were acquired during forward motion of the imaging probe, forexample, by highlighting portions of the image stack that were acquiredduring the forward motion.

For some applications, forward motion of the imaging probe is detectedby acquiring angiographic images or fluoroscopic images of the imagingprobe within the lumen, and performing image processing on theangiographic images in order to determine locations of the endoluminalimage probe marker with respect to the lumen at the time of theacquisition of respective endoluminal image frames, as describedhereinabove. Typically, as described hereinabove, image processing ofangiographic images can be used to identify forward motion of theimaging probe that is caused by (a) pulsation of the lumen, and (b) anoperator of the probe pushing the probe forward. Further typically,image processing of fluoroscopic images can only be used to identifyforward motion of the imaging probe that is caused by an operator of theprobe pushing the probe forward. For some applications, forward motionof the endoluminal probe that is caused by an operator pushing the probeforward is determined using a longitudinal position/movement sensorcoupled to apparatus through which the endoluminal probe is inserted,e.g., as described hereinabove with reference to FIG. 2.

For some applications, during pullback of the endoluminal imagingdevice, the subject's ECG signal is detected. Respective endoluminalimages are identified as corresponding to the period in the subject'scardiac cycle at the time when the image was acquired, based upon thedetected ECG signal (e.g., by indexing the image frames with respect tothe subject's ECG signal). For some applications, based upon theidentified correspondence, the system determines which of theendoluminal images were acquired in a given period of the subject'scardiac cycle, such as at least a portion of systole. Typically,associating endoluminal image frames with phases of the subject'scardiac cycle (e.g., by indexing with respect to the subject's ECGsignal) can be used to account for forward motion of the endoluminalimaging probe that is caused by motion of the probe with respect to thelumen due to pulsation of the lumen that is due to the subject's cardiaccycle.

For some applications, in order to generate the image stack it isdetermined which image frames were acquired during forward motion of theendoluminal imaging probe (e.g., based upon image processing ofangiographic or fluoroscopic images of the device inside the lumen, orbased upon associating the frames with respective phases of thesubject's cardiac cycle, such as, by indexing the frames with respect tothe subject's ECG signal), and, in response thereto, those image framesare either rejected, or are appropriately placed within the stack. Forsome applications, in order to generate the image stack it is determinedwhich locations along the lumen have two or more endoluminal imagescorresponding thereto, and, in response thereto, at least one of theimage frames corresponding to the location is rejected from being usedin the endoluminal image stack. Typically, only the first imaging frameto have been acquired at each location along the lumen is used in theimage stack, and the other image frames acquired at the location arerejected from being used in the image stack. Further typically, it isdetermined which at least one of the two or more endoluminal imageframes that correspond to the same location along the lumen wereacquired during forward motion of the probe, and this frame is rejectedfrom being used in the image stack. Alternatively or additionally,another at least one of the two or more endoluminal image frames thatcorrespond to the same location along the lumen is rejected from beingused in the image stack.

It is noted that some applications of the present invention have beendescribed with respect to an endoluminal image probe that acquires imageframes while moving generally in a distal to proximal direction (i.e.,during pullback of the imaging probe), but that experiences somemovement in a proximal to distal direction. The scope of the presentinvention includes applying the techniques described herein to anendoluminal image probe that acquires image frames while movinggenerally in a proximal to distal direction (i.e., while the probe isbeing pushed forward through the lumen), but that experiences somemovement in a distal to proximal direction, mutatis mutandis.

For some applications, techniques described herein (e.g., techniquesdescribed with reference to FIG. 12) are performed by a system thatincludes at least one processor, for use with an endoluminaldata-acquisition device that acquires a plurality of endoluminal datapoints of a lumen of a body of a subject while being moved through thelumen generally in a first direction with respect to the lumen. For someapplications, the processor includes (a)duplicate-data-point-identification functionality configured todetermine that, at at least one location, two or more endoluminal datapoints were acquired by the endoluminal data-acquisition device, (b)data-point-selection functionality configured to generate an outputusing a portion of the plurality of endoluminal data points of the lumenacquired using the endoluminal data-acquisition device, by using only asingle data point corresponding to the location, and (c) display-drivingfunctionality configured to drive a display to display the output.

For some applications, the processor includes (a)direction-determination functionality configured to determine that,while acquiring at least one of the endoluminal data points, theendoluminal data-acquisition device was moving in a second directionthat is opposite to the first direction, (b) output-generationfunctionality configured, in response to the determining, to generate anoutput using at least some of the plurality of endoluminal data pointsof the lumen acquired using the endoluminal data-acquisition device, and(c) display-driving functionality configured to drive a display todisplay the output.

For some applications, locations of an endoluminal imaging probeassociated with a first endoluminal modality (e.g., IVUS) are identifiedas corresponding to respective endoluminal image frames of the firstimaging modality, in accordance with the techniques describedhereinabove. Subsequently, locations of an endoluminal imaging probeassociated with a second endoluminal modality (e.g., OCT) are identifiedas corresponding to respective endoluminal image frames of the secondimaging modality, in accordance with the techniques describedhereinabove. For example, forward motion of one or both of theendoluminal imaging probes may be accounted for in associating thelocations of the endoluminal image probes with the image frames, inaccordance with techniques described hereinabove. Consequently, the twodata sets are co-registered to one another. For some applications, thetwo endoluminal data sets are displayed as overlaid or otherwise mergedwith one another.

For some applications, in order to determine the angular orientation ofthe probe with respect to the lumen at the time of the acquisition ofrespective endoluminal image frames, an asymmetrically shaped radiopaquemarker that is visible in extraluminal images (e.g., angiographic orfluoroscopic images) of the lumen is disposed on the imaging head of theendoluminal probe. Alternatively or additionally, the marker may bedisposed asymmetrically with respect to the longitudinal axis of theimaging head of the endoluminal probe. During the acquisition ofendoluminal image frames by the endoluminal imaging probe, extraluminalimages are acquired of the endoluminal image probe within the lumen.Image processing is applied to the fluoroscopic images in order todetermine the angular orientation of the probe with respect to the lumenat the time of the acquisition of respective endoluminal image frames,typically automatically and typically on-line, in accordance withtechniques described herein.

For some applications, the aforementioned techniques are applied inorder to account for unintentional rotation (typically, roll) of theendoluminal imaging probe with respect to the lumen, due to pulsation ofthe lumen, for example. For some applications, the aformentionedtechniques are applied in order to facilitate the generation of anendoluminal image stack, in which the images that comprise the stack arecorrectly rotationally aligned. Alternatively or additionally, theaforementioned techniques are applied to determine the orientation withrespect to each other of vessels that appear in the endoluminal images.

Reference is now made to FIG. 13 which shows image frames 120, 122, and124 of a stent inside a blood vessel. Frame 120 is a raw image frame ofthe stent inside the blood vessel.

For some applications, an enhanced extraluminal image, or imagesequence, of a deployed device and/or tool (for example, a stent) isdisplayed. In accordance with respective applications, the enhancedimage of the deployed device is co-registered to an endoluminal image(e.g., in accordance with the techniques described herein), or isdisplayed independently of any endoluminal images. For someapplications, the enhancement is performed in accordance with techniquesdescribed in US Patent Application 2010/0172556 to Cohen et al., whichis incorporated herein by reference.

For some applications, the image of the tool within the stabilized imagestream is enhanced in real time or near real time. For someapplications, enhancement of the image of the tool is performed incombination with the techniques described in WO 08/107905 to Iddan,which is incorporated herein by reference.

For some applications, enhancement is performed automatically uponframes that have been image-tracked such that the tool is displayed in asame or similar relative location throughout most or all frames, asdescribed in US Patent Application 2010/0172556 to Cohen, which isincorporated herein by reference. For some applications, enhancement isperformed by means of temporal filtering of the image-tracked frames.Typically, enhancement is performed in real time, or in near real time.Frame 122 of FIG. 13 is an enhanced image frame, generated in accordancewith techniques described in US Patent Application 2010/0172556 toCohen. It may be observed that stent 126 is more visible in frame 122than in raw image frame 120.

For some applications, the temporal filtering applies a weightedaveraging function to the value of each pixel, as defined by itsrelative locations in a series of consecutive frames, and displays theresulting image. Alternatively or additionally, the temporal filteringapplies a median function to the value of each pixel, as defined by itsrelative locations in a series of consecutive frames, and displays theresulting image. Further alternatively or additionally, the temporalfiltering applies a mode function to the value of each pixel, as definedby its relative locations in a series of consecutive frames, anddisplays the resulting image.

For some applications, in addition to the application of a temporalfilter, a spatial filter is applied to increase the contrast in theenhanced image. For example, the spatial filter may be a levelingfilter. For some applications, contrast is increased by histogramstretching, and/or by gamma correction.

In accordance with respective applications, contrast enhancement isspecifically applied to the edges of a tool, such as a balloon, or tothe struts of a tool, such as a stent.

For some applications, only the final image, representing the outcome ofthe enhancement process, is displayed. Alternatively, intermediateframes, reflecting gradual enhancement, are also displayed on-line.

For some applications, enhancement is performed upon a number oftypically-consecutive gated image frames. When using gated images, theenhancement is typically applied to fewer image frames than when theenhancement is applied to non-gated image frames, which may degrade theoutcome of the enhancement process. However, such gated frames are oftenalready aligned to a substantial extent, which may improve the outcomeof the enhancement process.

For some applications, alternative or additional techniques are appliedto enhance the visibility of the tool in an extraluminal image or imagestream, the techniques being performed typically on-line, and typicallyautomatically. For some applications, the enhancement is performed byapplying an iterative algorithm on a set of images, the algorithmoperating as follows. An initial enhanced image is calculated fromregistered images, typically by means of techniques disclosedhereinabove, such as temporal filtering. In each iteration, thealgorithm attempts to improve the already-created enhanced image, byselecting only some of the image frames to be used for creating a newenhanced image frame, and not using the remaining image frames.

Typically, device contours are identified in at least some of the imageframes from which a recent enhanced image, or image stream, wasgenerated by the system (typically automatically, and typically on-line)(a) identifying marker locations within the image frame, (b) identifyingcurved edge lines in the vicinity of markers, and (c) interpreting theedge lines as the device contours. From among those image frames, asubset of the image frames in which the device contours are most similarto each other is selected, and other image frames are rejected. It isnoted that in accordance with this technique, some image frames arerejected from the subset of image frames, even though edge linescorresponding to the device contours appear in the rejected imageframes.

For some applications, the similarity of the device contours in a set ofimage frames is determined based upon the similarity of the shapes ofthe edge lines in the image frames. Alternatively or additionally, thesimilarity of the device contours in a set of image frames is determinedby determining an extent to which the edge lines are parallel to animaginary line running from a first (e.g., distal) marker to a second(e.g., proximal) marker in the image frames.

Subsequent to the subset of image frames having been selected, thatsubset is used for creating a new enhanced image, again with theenhancement performed according to techniques disclosed hereinabove.Typically, before averaging the subset of image frames in order tocreate the new enhanced image frame, at least some of the image framesin the subset are translated, such that the edge lines in all of theimage frames in the subset are aligned with each other.

As described hereinabove, typically, in each step of the iterativealgorithm, the image frames in which the device contours are the mostsimilar to each other are selected. Alternatively, a single image frameis selected as a baseline image frame, and image frames are selectedbased upon a level of similarity of device contours in the image framesto those of the baseline image frame.

Typically, the above-described algorithm is applied iteratively until nomore image frames are excluded from the most recent subset. Typically,the final outcome of applying the iterative algorithm is an enhancedimage frame in which at least one of the device contour, or edges, orstruts, or other device elements, are more visible than they are innon-enhanced images frames, or in enhanced image frames that have nothad the above iterative algorithm applied to them, ceteris paribus.Typically, applying the iterative algorithm to image frames that havebeen enhanced in accordance with techniques described in US 2010/0172556to Cohen, which is incorporated herein by reference, further enhancesthe image frames.

For example, the iterative enhancement may be used when enhancing astent previously deployed by a balloon carrying radiopaque markers. Thedeflated balloon still resides, intraluminally, within the deployedstent. Typically at this stage, due to pulsation of the lumen, theballoon and the radiopaque markers thereof shift (e.g., axially shift,and/or radially shift) with respect to the endoluminal walls. The stentis fixated to the endoluminal walls, and does not therefore shift withrespect to the endoluminal walls. Consequently, generating an enhancedimage of the stent from all image frames acquired along the cardiaccycle, using a technique that relies upon using the locations of theradiopaque markers (e.g., as described in US Patent Application2010/0172556 to Cohen, which is incorporated herein by reference) mightsuffer from a blurring effect resulting from the different locations ofthe balloon markers, relative to the stent struts, in some of thoseframes.

The application of the iterative enhancement algorithm disclosedhereinabove, in which image frames are selected based upon thesimilarity of contours in the image frames to the device contours,typically reduces such a blurring effect. Thus, using theabove-described iterative enhancement algorithm for generating anenhanced image frame (according to which, some image frames are rejectedfrom being used to generate the enhanced image frame) may produce abetter-enhanced image of the deployed stent than an enhanced image framethat is generated using all the image frames (or all of the gated imageframes) irrespective of the similarity of contours in the image framesto the device contours.

Frame 124 of FIG. 13 is an enhanced image frame, generated in accordancewith techniques described in US Patent Application 2010/0172556 toCohen, and using the iterative algorithm, in accordance with someapplications of the present invention. It may be observed that stent 126is more visible in frame 124 than in raw image frame 120, and in imageframe 122, which was generated using only techniques described in USPatent Application 2010/0172556 to Cohen.

For some applications, an enhanced image stream is displayed, byenhancing a plurality of image frames using techniques described herein(e.g., using the above-described iterative algorithm), and displayingthe enhanced image frames as an image stream.

For some applications, techniques described herein (e.g., techniquesdescribed with reference to FIG. 13) are performed by a system thatincludes at least one processor. For some applications, the processorincludes image-receiving functionality configured to receive theplurality of image frames into the processor, and marker-identifyingfunctionality configured to automatically identify radiopaque markers inthe image frames. Typically, the processor further includesedge-line-identifying functionality configured to automatically identifyedge lines in a vicinity of the radiopaque markers in the image frames,and image-selection functionality configured, in response to theidentifying of the edge lines, to select a subset of the image framesthat are based upon the acquired image frames, based upon a level ofsimilarity between the edge lines in the selected image frames to oneanother. For some applications, the processor includes image-alignmentfunctionality configured to align the edge lines in a plurality of theselected image frames. Typically, the processor includes image-averagingfunctionality configured to generate an averaged image frame byaveraging the plurality of aligned image frames, and display-drivingfunctionality configured to drive a display to display the averagedimage frame.

It is noted that although some techniques for co-using extraluminalimages and endoluminal data are described hereinabove primarily withrespect to extraluminal fluoroscopic/angiographic images and endoluminalIVUS images, the scope of the present invention includes applying thetechniques described herein to other forms of extraluminal andendoluminal images and/or data, mutatis mutandis. For example, theextraluminal images may include images generated by fluoroscopy, CT,MRI, ultrasound, PET, SPECT, other extraluminal imaging techniques, orany combination thereof. Endoluminal images may include images generatedby optical coherence tomography (OCT), near-infrared spectroscopy(NIRS), intravascular ultrasound (IVUS), endobronchial ultrasound(EBUS), magnetic resonance (MR), other endoluminal imaging techniques,or any combination thereof. Endoluminal data may include data related topressure (e.g., fractional flow reserve), flow, temperature, electricalactivity, or any combination thereof. Examples of the anatomicalstructure to which the aforementioned co-registration of extraluminaland endoluminal images may be applied include a coronary vessel, acoronary lesion, a vessel, a vascular lesion, a lumen, a luminal lesion,and/or a valve. It is noted that the scope of the present inventionincludes applying the techniques described herein to lumens of asubject's body other than blood vessels (for example, a lumen of thegastrointestinal or respiratory tract).

It will be appreciated by persons skilled in the art that the presentinvention is not limited to what has been particularly shown anddescribed hereinabove. Rather, the scope of the present inventionincludes both combinations and subcombinations of the various featuresdescribed hereinabove, as well as variations and modifications thereofthat are not in the prior art, which would occur to persons skilled inthe art upon reading the foregoing description.

1. A co-registration system, comprising: a processor configured for communication with an extraluminal imaging device, a first endoluminal data-acquisition device, a different second endoluminal data-acquisition device, and a display, wherein the processor is configured to: receive an extraluminal image of a lumen of a body of a subject; receive, from the first endoluminal data-acquisition device operating according to a first modality and positioned at a first location within the lumen, first endoluminal data associated with the lumen at the first location, wherein the first endoluminal data comprises data of the first modality; associate the first endoluminal data with a first corresponding location of the lumen in the extraluminal image; receive, from the second endoluminal data-acquisition device operating according to a different second modality and positioned at a second location within the lumen, second endoluminal data associated with the lumen at the second location, wherein the second endoluminal data comprises data of the second modality; associate the second endoluminal data with a second corresponding location of the lumen in the extraluminal image; generate, based on associating the first endoluminal data with the first corresponding location and associating the second endoluminal data with the second corresponding location, a screen display comprising: the extraluminal image; a first indication of the first endoluminal data at the first corresponding location in the extraluminal image; and a second indication of the second endoluminal data at the second corresponding location in the extraluminal image; and output the screen display to the display.
 2. The co-registration system of claim 1, wherein the first location is different from the second location.
 3. The co-registration system of claim 1, wherein the first endoluminal data comprises an endoluminal image of the lumen at the first location.
 4. The co-registration system of claim 3, wherein the endoluminal image comprises an intravascular ultrasound image or an optical coherence tomography image.
 5. The co-registration system of claim 3, wherein the second endoluminal data comprises at least one of pressure, flow, or temperature within the lumen at the second location.
 6. The co-registration system of claim 3, wherein the screen display further comprises: an endoluminal image stack comprising the endoluminal image; and a marking associated with the first location in the endoluminal image stack.
 7. The co-registration system of claim 1, wherein the first indication and the second indication are overlaid on the extraluminal image in the screen display.
 8. The co-registration system of claim 1, wherein the screen display further comprises at least one of the first endoluminal data or the second endoluminal data overlaid on the extraluminal image.
 9. The co-registration system of claim 1, wherein the processor is configured to merge the first endoluminal data and the second endoluminal data.
 10. The co-registration system of claim 1, further comprising the display.
 11. The co-registration system of claim 1, further comprising the extraluminal imaging device.
 12. The co-registration system of claim 1, further comprising the first endoluminal data-acquisition device.
 13. The co-registration system of claim 1, further comprising the second endoluminal data-acquisition device. 